Oct 30 16:44:10.694433 ip-10-0-131-13 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 16:44:10.694447 ip-10-0-131-13 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 16:44:10.694457 ip-10-0-131-13 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 16:44:10.694886 ip-10-0-131-13 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 16:44:20.896086 ip-10-0-131-13 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 16:44:20.896100 ip-10-0-131-13 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot f25f24616b04403694dbce6afe849079 --
Oct 30 16:46:23.125729 ip-10-0-131-13 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 16:46:23.429805 ip-10-0-131-13 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 16:46:23.429805 ip-10-0-131-13 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 16:46:23.429805 ip-10-0-131-13 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 16:46:23.429805 ip-10-0-131-13 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 16:46:23.433618 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433532    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 16:46:23.433618 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433618    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 16:46:23.433618 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433622    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 16:46:23.433723 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433625    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 16:46:23.433723 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433628    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 16:46:23.433723 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433631    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 16:46:23.433723 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433634    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 16:46:23.433723 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433637    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 16:46:23.433723 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433640    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 16:46:23.433723 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433642    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 16:46:23.433723 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433645    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 16:46:23.433723 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433649    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 16:46:23.433723 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433653    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 16:46:23.433723 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433656    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 16:46:23.433723 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433658    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 16:46:23.433723 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433661    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 16:46:23.433723 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433664    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 16:46:23.433723 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433667    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 16:46:23.433723 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433670    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 16:46:23.433723 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433681    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 16:46:23.433723 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433684    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 16:46:23.433723 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433687    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 16:46:23.434247 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433689    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 16:46:23.434247 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433692    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 16:46:23.434247 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433695    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 16:46:23.434247 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433697    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 16:46:23.434247 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433700    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 16:46:23.434247 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433704    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 16:46:23.434247 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433711    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 16:46:23.434247 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433716    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 16:46:23.434247 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433721    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 16:46:23.434247 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433726    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 16:46:23.434247 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433731    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 16:46:23.434247 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433735    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 16:46:23.434247 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433739    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 16:46:23.434247 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433741    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 16:46:23.434247 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433744    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 16:46:23.434247 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433747    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 16:46:23.434247 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433749    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 16:46:23.434247 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433752    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 16:46:23.434247 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433754    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 16:46:23.434247 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433757    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 16:46:23.434734 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433760    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 16:46:23.434734 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433762    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 16:46:23.434734 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433766    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 16:46:23.434734 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433770    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 16:46:23.434734 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433773    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 16:46:23.434734 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433777    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 16:46:23.434734 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433780    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 16:46:23.434734 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433782    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 16:46:23.434734 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433785    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 16:46:23.434734 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433787    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 16:46:23.434734 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433790    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 16:46:23.434734 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433792    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 16:46:23.434734 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433795    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 16:46:23.434734 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433799    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 16:46:23.434734 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433802    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 16:46:23.434734 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433804    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 16:46:23.434734 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433807    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 16:46:23.434734 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433811    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 16:46:23.434734 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433813    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 16:46:23.435186 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433816    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 16:46:23.435186 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433820    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 16:46:23.435186 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433822    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 16:46:23.435186 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433825    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 16:46:23.435186 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433827    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 16:46:23.435186 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433830    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 16:46:23.435186 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433833    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 16:46:23.435186 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433835    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 16:46:23.435186 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433838    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 16:46:23.435186 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433840    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 16:46:23.435186 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433843    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 16:46:23.435186 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433846    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 16:46:23.435186 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433849    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 16:46:23.435186 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433851    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 16:46:23.435186 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433854    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 16:46:23.435186 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433857    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 16:46:23.435186 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433860    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 16:46:23.435186 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433863    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 16:46:23.435186 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433866    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 16:46:23.435186 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433869    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 16:46:23.435671 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433872    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 16:46:23.435671 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433875    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 16:46:23.435671 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433878    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 16:46:23.435671 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433880    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 16:46:23.435671 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433883    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 16:46:23.435671 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433885    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 16:46:23.435671 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433889    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 16:46:23.435671 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433891    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 16:46:23.435671 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433894    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 16:46:23.435671 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433897    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 16:46:23.435671 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433899    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 16:46:23.435671 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433902    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 16:46:23.435671 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433905    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 16:46:23.435671 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433907    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 16:46:23.435671 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433910    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 16:46:23.435671 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433913    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 16:46:23.435671 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433915    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 16:46:23.435671 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433918    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 16:46:23.435671 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433920    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 16:46:23.436120 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433930    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 16:46:23.436120 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433933    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 16:46:23.436120 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433936    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 16:46:23.436120 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433940    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 16:46:23.436120 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433943    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 16:46:23.436120 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.433945    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 16:46:23.436120 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434772    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 16:46:23.436120 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434779    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 16:46:23.436120 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434782    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 16:46:23.436120 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434785    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 16:46:23.436120 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434788    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 16:46:23.436120 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434790    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 16:46:23.436120 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434793    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 16:46:23.436120 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434796    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 16:46:23.436120 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434799    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 16:46:23.436120 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434802    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 16:46:23.436120 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434804    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 16:46:23.436120 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434807    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 16:46:23.436120 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434812    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 16:46:23.436604 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434818    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 16:46:23.436604 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434821    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 16:46:23.436604 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434824    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 16:46:23.436604 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434827    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 16:46:23.436604 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434830    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 16:46:23.436604 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434833    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 16:46:23.436604 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434835    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 16:46:23.436604 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434838    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 16:46:23.436604 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434841    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 16:46:23.436604 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434843    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 16:46:23.436604 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434846    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 16:46:23.436604 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434849    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 16:46:23.436604 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434851    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 16:46:23.436604 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434854    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 16:46:23.436604 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434857    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 16:46:23.436604 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434859    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 16:46:23.436604 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434862    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 16:46:23.436604 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434864    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 16:46:23.436604 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434867    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 16:46:23.436604 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434869    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 16:46:23.437087 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434872    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 16:46:23.437087 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434875    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 16:46:23.437087 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434877    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 16:46:23.437087 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434880    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 16:46:23.437087 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434883    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 16:46:23.437087 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434886    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 16:46:23.437087 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434889    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 16:46:23.437087 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434892    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 16:46:23.437087 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434895    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 16:46:23.437087 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434897    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 16:46:23.437087 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434900    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 16:46:23.437087 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434902    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 16:46:23.437087 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434905    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 16:46:23.437087 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434908    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 16:46:23.437087 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434912    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 16:46:23.437087 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434916    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 16:46:23.437087 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434919    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 16:46:23.437087 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434921    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 16:46:23.437087 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434924    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 16:46:23.437557 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434926    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 16:46:23.437557 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434929    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 16:46:23.437557 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434931    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 16:46:23.437557 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434934    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 16:46:23.437557 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434936    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 16:46:23.437557 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434939    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 16:46:23.437557 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434942    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 16:46:23.437557 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434944    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 16:46:23.437557 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434947    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 16:46:23.437557 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434950    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 16:46:23.437557 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434952    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 16:46:23.437557 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434955    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 16:46:23.437557 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434957    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 16:46:23.437557 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434960    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 16:46:23.437557 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434962    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 16:46:23.437557 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434965    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 16:46:23.437557 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434968    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 16:46:23.437557 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434971    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 16:46:23.437557 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434973    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 16:46:23.437557 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434976    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 16:46:23.438046 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434978    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 16:46:23.438046 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434981    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 16:46:23.438046 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434984    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 16:46:23.438046 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434986    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 16:46:23.438046 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434989    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 16:46:23.438046 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434992    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 16:46:23.438046 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434994    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 16:46:23.438046 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.434997    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 16:46:23.438046 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435000    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 16:46:23.438046 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435003    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 16:46:23.438046 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435005    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 16:46:23.438046 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435008    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 16:46:23.438046 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435011    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 16:46:23.438046 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435013    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 16:46:23.438046 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435016    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 16:46:23.438046 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435018    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 16:46:23.438046 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435021    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 16:46:23.438046 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435023    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 16:46:23.438046 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435026    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 16:46:23.438046 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435028    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 16:46:23.438528 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435031    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 16:46:23.438528 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435033    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 16:46:23.438528 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435035    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 16:46:23.438528 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435038    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 16:46:23.438528 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435041    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 16:46:23.438528 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435043    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 16:46:23.438528 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435046    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 16:46:23.438528 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435048    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 16:46:23.438528 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435051    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 16:46:23.438528 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435054    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 16:46:23.438528 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435056    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 16:46:23.438528 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435059    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 16:46:23.438528 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435062    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 16:46:23.438528 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435065    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 16:46:23.438528 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435611    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 16:46:23.438528 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435618    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 16:46:23.438528 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435622    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 16:46:23.438528 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435625    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 16:46:23.438528 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435628    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 16:46:23.439014 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435631    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 16:46:23.439014 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435634    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 16:46:23.439014 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435636    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 16:46:23.439014 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435639    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 16:46:23.439014 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435642    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 16:46:23.439014 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435645    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 16:46:23.439014 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435648    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 16:46:23.439014 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435650    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 16:46:23.439014 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435653    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 16:46:23.439014 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435656    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 16:46:23.439014 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435658    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 16:46:23.439014 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435661    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 16:46:23.439014 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435663    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 16:46:23.439014 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435666    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 16:46:23.439014 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435669    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 16:46:23.439014 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435671    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 16:46:23.439014 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435674    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 16:46:23.439014 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435678    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 16:46:23.439014 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435680    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 16:46:23.439014 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435683    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 16:46:23.439508 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435686    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 16:46:23.439508 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435689    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 16:46:23.439508 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435691    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 16:46:23.439508 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435694    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 16:46:23.439508 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435696    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 16:46:23.439508 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435699    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 16:46:23.439508 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435701    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 16:46:23.439508 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435708    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 16:46:23.439508 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435711    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 16:46:23.439508 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435714    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 16:46:23.439508 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435716    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 16:46:23.439508 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435718    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 16:46:23.439508 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435722    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 16:46:23.439508 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435726    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 16:46:23.439508 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435729    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 16:46:23.439508 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435732    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 16:46:23.439508 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435735    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 16:46:23.439508 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435738    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 16:46:23.439508 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435741    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 16:46:23.439953 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435743    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 16:46:23.439953 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435746    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 16:46:23.439953 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435749    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 16:46:23.439953 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435751    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 16:46:23.439953 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435754    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 16:46:23.439953 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435756    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 16:46:23.439953 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435759    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 16:46:23.439953 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435762    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 16:46:23.439953 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435764    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 16:46:23.439953 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435767    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 16:46:23.439953 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435770    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 16:46:23.439953 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435772    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 16:46:23.439953 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435775    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 16:46:23.439953 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435777    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 16:46:23.439953 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435780    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 16:46:23.439953 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435783    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 16:46:23.439953 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435786    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 16:46:23.439953 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435789    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 16:46:23.439953 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435792    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 16:46:23.440459 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435794    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 16:46:23.440459 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435797    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 16:46:23.440459 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435800    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 16:46:23.440459 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435802    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 16:46:23.440459 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435805    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 16:46:23.440459 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435807    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 16:46:23.440459 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435810    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 16:46:23.440459 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435812    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 16:46:23.440459 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435815    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 16:46:23.440459 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435817    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 16:46:23.440459 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435820    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 16:46:23.440459 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435823    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 16:46:23.440459 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435825    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 16:46:23.440459 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435827    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 16:46:23.440459 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435830    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 16:46:23.440459 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435832    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 16:46:23.440459 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435835    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 16:46:23.440459 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435837    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 16:46:23.440459 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435840    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 16:46:23.440459 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435842    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 16:46:23.440926 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435845    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 16:46:23.440926 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435847    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 16:46:23.440926 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435850    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 16:46:23.440926 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435856    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 16:46:23.440926 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435858    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 16:46:23.440926 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435861    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 16:46:23.440926 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435863    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 16:46:23.440926 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435866    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 16:46:23.440926 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435869    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 16:46:23.440926 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435871    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 16:46:23.440926 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435874    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 16:46:23.440926 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435877    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 16:46:23.440926 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435879    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 16:46:23.440926 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435882    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 16:46:23.440926 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435884    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 16:46:23.440926 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435887    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 16:46:23.440926 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435890    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 16:46:23.440926 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435893    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 16:46:23.440926 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435895    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 16:46:23.440926 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435898    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 16:46:23.441392 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435903    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 16:46:23.441392 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435906    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 16:46:23.441392 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.435910    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 16:46:23.441392 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436664    2581 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 16:46:23.441392 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436675    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 16:46:23.441392 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436682    2581 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 16:46:23.441392 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436688    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 16:46:23.441392 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436692    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 16:46:23.441392 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436696    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 16:46:23.441392 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436700    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 16:46:23.441392 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436705    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 16:46:23.441392 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436708    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 16:46:23.441392 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436712    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 16:46:23.441392 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436715    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 16:46:23.441392 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436719    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 16:46:23.441392 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436724    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 16:46:23.441392 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436728    2581 flags.go:64] FLAG: --cgroup-root=""
Oct 30 16:46:23.441392 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436730    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 16:46:23.441392 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436734    2581 flags.go:64] FLAG: --client-ca-file=""
Oct 30 16:46:23.441392 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436737    2581 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 16:46:23.441392 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436740    2581 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 16:46:23.441392 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436746    2581 flags.go:64] FLAG: --cluster-domain=""
Oct 30 16:46:23.441392 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436749    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 16:46:23.441978 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436753    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 16:46:23.441978 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436756    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 16:46:23.441978 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436759    2581 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 16:46:23.441978 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436764    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 16:46:23.441978 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436767    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 16:46:23.441978 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436771    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 16:46:23.441978 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436775    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 16:46:23.441978 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436778    2581 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 16:46:23.441978 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436782    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 16:46:23.441978 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436785    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 16:46:23.441978 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436788    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 16:46:23.441978 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436791    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 16:46:23.441978 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436795    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 16:46:23.441978 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436798    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 16:46:23.441978 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436802    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 16:46:23.441978 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436805    2581 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 16:46:23.441978 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436808    2581 flags.go:64] FLAG: --enable-server="true"
Oct 30 16:46:23.441978 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436811    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 16:46:23.441978 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436816    2581 flags.go:64] FLAG: --event-burst="100"
Oct 30 16:46:23.441978 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436819    2581 flags.go:64] FLAG: --event-qps="50"
Oct 30 16:46:23.441978 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436822    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 16:46:23.441978 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436825    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 16:46:23.441978 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436828    2581 flags.go:64] FLAG: --eviction-hard=""
Oct 30 16:46:23.441978 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436833    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 16:46:23.442567 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436837    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 16:46:23.442567 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436841    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 16:46:23.442567 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436844    2581 flags.go:64] FLAG: --eviction-soft=""
Oct 30 16:46:23.442567 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436847    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 16:46:23.442567 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436856    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 16:46:23.442567 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436859    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 16:46:23.442567 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436862    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 16:46:23.442567 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436866    2581 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 16:46:23.442567 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436869    2581 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 16:46:23.442567 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436872    2581 flags.go:64] FLAG: --feature-gates=""
Oct 30 16:46:23.442567 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436876    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 16:46:23.442567 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436879    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 16:46:23.442567 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436883    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 16:46:23.442567 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436886    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 16:46:23.442567 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436891    2581 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 16:46:23.442567 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436894    2581 flags.go:64] FLAG: --help="false"
Oct 30 16:46:23.442567 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436898    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-131-13.ec2.internal"
Oct 30 16:46:23.442567 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436901    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 16:46:23.442567 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436904    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 16:46:23.442567 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436907    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 16:46:23.442567 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436910    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 16:46:23.442567 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436914    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 16:46:23.442567 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436917    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 16:46:23.442567 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436920    2581 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 16:46:23.443212 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436923    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 16:46:23.443212 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436926    2581 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 16:46:23.443212 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436929    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 16:46:23.443212 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436932    2581 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 16:46:23.443212 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436935    2581 flags.go:64] FLAG: --kube-reserved=""
Oct 30 16:46:23.443212 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436938    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 16:46:23.443212 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436941    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 16:46:23.443212 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436944    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 16:46:23.443212 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436949    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 16:46:23.443212 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436952    2581 flags.go:64] FLAG: --lock-file=""
Oct 30 16:46:23.443212 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436955    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 16:46:23.443212 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436959    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 16:46:23.443212 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436962    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 16:46:23.443212 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436968    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 16:46:23.443212 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436971    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 16:46:23.443212 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436974    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 16:46:23.443212 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436977    2581 flags.go:64] FLAG: --logging-format="text"
Oct 30 16:46:23.443212 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436981    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 16:46:23.443212 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436984    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 16:46:23.443212 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436987    2581 flags.go:64] FLAG: --manifest-url=""
Oct 30 16:46:23.443212 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436990    2581 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 16:46:23.443212 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436995    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 16:46:23.443212 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.436998    2581 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 16:46:23.443212 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437002    2581 flags.go:64] FLAG: --max-pods="110"
Oct 30 16:46:23.443212 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437005    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 16:46:23.444136 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437008    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 16:46:23.444136 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437011    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 16:46:23.444136 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437015    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 16:46:23.444136 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437018    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 16:46:23.444136 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437020    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 16:46:23.444136 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437023    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 16:46:23.444136 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437031    2581 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 16:46:23.444136 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437035    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 16:46:23.444136 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437038    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 16:46:23.444136 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437041    2581 flags.go:64] FLAG: --pod-cidr=""
Oct 30 16:46:23.444136 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437044    2581 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 16:46:23.444136 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437047    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 16:46:23.444136 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437050    2581 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 16:46:23.444136 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437054    2581 flags.go:64] FLAG: --port="10250"
Oct 30 16:46:23.444136 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437057    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 16:46:23.444136 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437061    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-047c5dfc4d5a57b81"
Oct 30 16:46:23.444136 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437065    2581 flags.go:64] FLAG: --qos-reserved=""
Oct 30 16:46:23.444136 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437068    2581 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 16:46:23.444136 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437073    2581 flags.go:64] FLAG: --register-node="true"
Oct 30 16:46:23.444136 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437076    2581 flags.go:64] FLAG: --register-with-taints=""
Oct 30 16:46:23.444136 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437083    2581 flags.go:64] FLAG: --registry-burst="10"
Oct 30 16:46:23.444136 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437086    2581 flags.go:64] FLAG: --registry-qps="5"
Oct 30 16:46:23.444136 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437089    2581 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 16:46:23.444136 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437093    2581 flags.go:64] FLAG: --reserved-memory=""
Oct 30 16:46:23.444136 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437097    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 16:46:23.444136 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437101    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 16:46:23.444858 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437104    2581 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 16:46:23.444858 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437107    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 16:46:23.444858 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437110    2581 flags.go:64] FLAG: --runonce="false"
Oct 30 16:46:23.444858 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437113    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 16:46:23.444858 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437116    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 16:46:23.444858 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437120    2581 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 16:46:23.444858 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437123    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 16:46:23.444858 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437126    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 16:46:23.444858 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437129    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 16:46:23.444858 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437133    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 16:46:23.444858 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437136    2581 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 16:46:23.444858 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437139    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 16:46:23.444858 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437143    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 16:46:23.444858 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437145    2581 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 16:46:23.444858 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437148    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 16:46:23.444858 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437151    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 16:46:23.444858 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437155    2581 flags.go:64] FLAG: --system-cgroups=""
Oct 30 16:46:23.444858 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437158    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 16:46:23.444858 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437163    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 16:46:23.444858 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437166    2581 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 16:46:23.444858 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437169    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 16:46:23.444858 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437175    2581 flags.go:64] FLAG: --tls-min-version=""
Oct 30 16:46:23.444858 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437179    2581 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 16:46:23.444858 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437182    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 16:46:23.444858 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437186    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 16:46:23.445473 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437189    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 16:46:23.445473 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437192    2581 flags.go:64] FLAG: --v="2"
Oct 30 16:46:23.445473 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437198    2581 flags.go:64] FLAG: --version="false"
Oct 30 16:46:23.445473 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437202    2581 flags.go:64] FLAG: --vmodule=""
Oct 30 16:46:23.445473 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437206    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 16:46:23.445473 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437210    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 16:46:23.445473 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437348    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 16:46:23.445473 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437351    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 16:46:23.445473 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437354    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 16:46:23.445473 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437357    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 16:46:23.445473 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437360    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 16:46:23.445473 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437363    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 16:46:23.445473 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437366    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 16:46:23.445473 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437369    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 16:46:23.445473 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437372    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 16:46:23.445473 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437375    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 16:46:23.445473 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437378    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 16:46:23.445473 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437381    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 16:46:23.445473 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437384    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 16:46:23.445473 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437387    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 16:46:23.445473 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437390    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 16:46:23.446007 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437392    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 16:46:23.446007 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437395    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 16:46:23.446007 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437398    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 16:46:23.446007 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437411    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 16:46:23.446007 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437414    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 16:46:23.446007 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437417    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 16:46:23.446007 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437420    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 16:46:23.446007 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437425    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 16:46:23.446007 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437427    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 16:46:23.446007 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437430    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 16:46:23.446007 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437434    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 16:46:23.446007 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437437    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 16:46:23.446007 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437440    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 16:46:23.446007 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437443    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 16:46:23.446007 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437446    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 16:46:23.446007 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437449    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 16:46:23.446007 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437452    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 16:46:23.446007 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437455    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 16:46:23.446007 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437457    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 16:46:23.446007 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437460    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 16:46:23.446518 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437463    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 16:46:23.446518 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437466    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 16:46:23.446518 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437468    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 16:46:23.446518 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437472    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 16:46:23.446518 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437475    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 16:46:23.446518 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437478    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 16:46:23.446518 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437481    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 16:46:23.446518 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437484    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 16:46:23.446518 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437486    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 16:46:23.446518 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437489    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 16:46:23.446518 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437492    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 16:46:23.446518 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437494    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 16:46:23.446518 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437497    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 16:46:23.446518 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437500    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 16:46:23.446518 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437503    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 16:46:23.446518 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437507    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 16:46:23.446518 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437510    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 16:46:23.446518 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437514    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 16:46:23.446518 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437517    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 16:46:23.446969 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437521    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 16:46:23.446969 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437524    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 16:46:23.446969 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437526    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 16:46:23.446969 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437530    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 16:46:23.446969 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437533    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 16:46:23.446969 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437536    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 16:46:23.446969 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437539    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 16:46:23.446969 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437541    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 16:46:23.446969 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437544    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 16:46:23.446969 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437547    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 16:46:23.446969 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437550    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 16:46:23.446969 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437552    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 16:46:23.446969 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437555    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 16:46:23.446969 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437558    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 16:46:23.446969 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437561    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 16:46:23.446969 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437564    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 16:46:23.446969 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437566    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 16:46:23.446969 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437569    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 16:46:23.446969 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437572    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 16:46:23.446969 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437576    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 16:46:23.447465 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437578    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 16:46:23.447465 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437581    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 16:46:23.447465 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437584    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 16:46:23.447465 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437587    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 16:46:23.447465 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437589    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 16:46:23.447465 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437592    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 16:46:23.447465 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437595    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 16:46:23.447465 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437597    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 16:46:23.447465 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437600    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 16:46:23.447465 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437602    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 16:46:23.447465 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437605    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 16:46:23.447465 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437608    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 16:46:23.447465 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437611    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 16:46:23.447465 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437613    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 16:46:23.447465 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437618    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 16:46:23.447465 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437622    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 16:46:23.447465 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437625    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 16:46:23.447465 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437628    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 16:46:23.447465 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437631    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 16:46:23.447465 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437633    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 16:46:23.447952 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437636    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 16:46:23.447952 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437639    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 16:46:23.447952 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437641    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 16:46:23.447952 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437644    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 16:46:23.447952 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437647    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 16:46:23.447952 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437649    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 16:46:23.447952 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437652    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 16:46:23.447952 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437654    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 16:46:23.447952 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437657    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 16:46:23.447952 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437660    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 16:46:23.447952 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437663    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 16:46:23.447952 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437665    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 16:46:23.447952 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.437693    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 16:46:23.447952 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437730    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 16:46:23.447952 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437735    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 16:46:23.447952 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437738    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 16:46:23.447952 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437741    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 16:46:23.448362 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437744    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 16:46:23.448362 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437747    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 16:46:23.448362 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437749    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 16:46:23.448362 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437752    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 16:46:23.448362 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437755    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 16:46:23.448362 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437757    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 16:46:23.448362 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437760    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 16:46:23.448362 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437763    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 16:46:23.448362 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437766    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 16:46:23.448362 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437768    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 16:46:23.448362 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437773    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 16:46:23.448362 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437776    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 16:46:23.448362 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437778    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 16:46:23.448362 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437781    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 16:46:23.448362 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437784    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 16:46:23.448362 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437787    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 16:46:23.448362 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437790    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 16:46:23.448362 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437792    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 16:46:23.448362 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437795    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 16:46:23.448362 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437798    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 16:46:23.448857 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437800    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 16:46:23.448857 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437803    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 16:46:23.448857 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437806    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 16:46:23.448857 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437809    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 16:46:23.448857 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437813    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 16:46:23.448857 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437816    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 16:46:23.448857 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437819    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 16:46:23.448857 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437822    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 16:46:23.448857 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437825    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 16:46:23.448857 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437828    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 16:46:23.448857 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437831    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 16:46:23.448857 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437834    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 16:46:23.448857 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437836    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 16:46:23.448857 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437839    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 16:46:23.448857 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437841    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 16:46:23.448857 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437844    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 16:46:23.448857 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437847    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 16:46:23.448857 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437849    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 16:46:23.448857 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437852    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 16:46:23.448857 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437855    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 16:46:23.449335 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437858    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 16:46:23.449335 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437861    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 16:46:23.449335 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437865    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 16:46:23.449335 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437867    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 16:46:23.449335 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437870    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 16:46:23.449335 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437873    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 16:46:23.449335 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437876    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 16:46:23.449335 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437878    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 16:46:23.449335 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437881    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 16:46:23.449335 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437883    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 16:46:23.449335 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437886    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 16:46:23.449335 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437889    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 16:46:23.449335 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437891    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 16:46:23.449335 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437894    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 16:46:23.449335 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437897    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 16:46:23.449335 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437900    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 16:46:23.449335 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437902    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 16:46:23.449335 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437905    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 16:46:23.449335 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437907    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 16:46:23.449813 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437910    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 16:46:23.449813 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437913    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 16:46:23.449813 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437916    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 16:46:23.449813 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437918    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 16:46:23.449813 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437921    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 16:46:23.449813 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437924    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 16:46:23.449813 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437926    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 16:46:23.449813 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437929    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 16:46:23.449813 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437931    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 16:46:23.449813 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437934    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 16:46:23.449813 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437937    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 16:46:23.449813 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437940    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 16:46:23.449813 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437943    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 16:46:23.449813 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437946    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 16:46:23.449813 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437948    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 16:46:23.449813 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437952    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 16:46:23.449813 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437955    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 16:46:23.449813 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437958    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 16:46:23.449813 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437961    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 16:46:23.450272 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437964    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 16:46:23.450272 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437966    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 16:46:23.450272 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437969    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 16:46:23.450272 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437972    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 16:46:23.450272 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437974    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 16:46:23.450272 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437977    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 16:46:23.450272 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437979    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 16:46:23.450272 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437982    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 16:46:23.450272 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437985    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 16:46:23.450272 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437988    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 16:46:23.450272 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437990    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 16:46:23.450272 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437993    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 16:46:23.450272 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.437996    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 16:46:23.450272 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.438000    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 16:46:23.450272 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.438004    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 16:46:23.450272 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.438007    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 16:46:23.450272 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.438010    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 16:46:23.450272 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.438014    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 16:46:23.450272 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.438017    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 16:46:23.450911 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.438019    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 16:46:23.450911 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.438022    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 16:46:23.450911 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.438025    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 16:46:23.450911 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.438028    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 16:46:23.450911 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.438031    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 16:46:23.450911 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.445773    2581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 16:46:23.450911 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.445791    2581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 16:46:23.450911 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.445806    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 16:46:23.450911 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.445812    2581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 16:46:23.450911 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445873    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 16:46:23.450911 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445879    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 16:46:23.450911 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445882    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 16:46:23.450911 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445885    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 16:46:23.450911 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445888    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 16:46:23.450911 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445892    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 16:46:23.450911 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445894    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 16:46:23.450911 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445897    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 16:46:23.450911 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445901    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 16:46:23.450911 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445905    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 16:46:23.451394 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445909    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 16:46:23.451394 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445912    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 16:46:23.451394 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445915    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 16:46:23.451394 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445917    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 16:46:23.451394 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445920    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 16:46:23.451394 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445923    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 16:46:23.451394 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445932    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 16:46:23.451394 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445935    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 16:46:23.451394 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445938    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 16:46:23.451394 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445941    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 16:46:23.451394 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445943    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 16:46:23.451394 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445946    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 16:46:23.451394 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445949    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 16:46:23.451394 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445952    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 16:46:23.451394 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445954    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 16:46:23.451394 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445957    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 16:46:23.451394 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445960    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 16:46:23.451394 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445963    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 16:46:23.451394 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445966    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 16:46:23.451915 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445969    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 16:46:23.451915 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445972    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 16:46:23.451915 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445975    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 16:46:23.451915 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445978    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 16:46:23.451915 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445980    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 16:46:23.451915 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445983    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 16:46:23.451915 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445987    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 16:46:23.451915 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445991    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 16:46:23.451915 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445994    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 16:46:23.451915 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445997    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 16:46:23.451915 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.445999    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 16:46:23.451915 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446002    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 16:46:23.451915 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446005    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 16:46:23.451915 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446007    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 16:46:23.451915 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446010    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 16:46:23.451915 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446013    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 16:46:23.451915 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446016    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 16:46:23.451915 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446020    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 16:46:23.451915 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446023    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 16:46:23.451915 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446026    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 16:46:23.452418 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446029    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 16:46:23.452418 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446032    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 16:46:23.452418 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446034    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 16:46:23.452418 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446037    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 16:46:23.452418 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446040    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 16:46:23.452418 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446042    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 16:46:23.452418 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446045    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 16:46:23.452418 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446048    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 16:46:23.452418 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446051    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 16:46:23.452418 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446053    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 16:46:23.452418 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446056    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 16:46:23.452418 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446059    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 16:46:23.452418 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446062    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 16:46:23.452418 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446065    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 16:46:23.452418 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446068    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 16:46:23.452418 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446071    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 16:46:23.452418 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446073    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 16:46:23.452418 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446076    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 16:46:23.452418 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446079    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 16:46:23.452418 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446081    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 16:46:23.452900 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446084    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 16:46:23.452900 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446086    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 16:46:23.452900 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446089    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 16:46:23.452900 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446091    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 16:46:23.452900 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446094    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 16:46:23.452900 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446096    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 16:46:23.452900 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446099    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 16:46:23.452900 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446101    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 16:46:23.452900 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446104    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 16:46:23.452900 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446106    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 16:46:23.452900 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446110    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 16:46:23.452900 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446112    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 16:46:23.452900 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446115    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 16:46:23.452900 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446118    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 16:46:23.452900 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446121    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 16:46:23.452900 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446123    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 16:46:23.452900 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446126    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 16:46:23.452900 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446128    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 16:46:23.452900 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446131    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 16:46:23.453331 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446134    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 16:46:23.453331 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446137    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 16:46:23.453331 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446139    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 16:46:23.453331 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446142    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 16:46:23.453331 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446145    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 16:46:23.453331 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446147    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 16:46:23.453331 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446150    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 16:46:23.453331 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446153    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 16:46:23.453331 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446155    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 16:46:23.453331 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446158    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 16:46:23.453331 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446160    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 16:46:23.453331 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446163    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 16:46:23.453331 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446166    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 16:46:23.453331 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446169    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 16:46:23.453331 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446171    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 16:46:23.453331 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446174    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 16:46:23.453331 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446176    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 16:46:23.453331 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446179    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 16:46:23.453764 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.446207    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 16:46:23.453764 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446355    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 16:46:23.453764 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446361    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 16:46:23.453764 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446364    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 16:46:23.453764 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446367    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 16:46:23.453764 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446371    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 16:46:23.453764 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446374    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 16:46:23.453764 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446377    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 16:46:23.453764 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446379    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 16:46:23.453764 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446382    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 16:46:23.453764 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446385    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 16:46:23.453764 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446388    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 16:46:23.453764 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446391    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 16:46:23.453764 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446394    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 16:46:23.453764 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446397    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 16:46:23.454170 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446400    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 16:46:23.454170 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446420    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 16:46:23.454170 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446423    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 16:46:23.454170 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446426    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 16:46:23.454170 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446429    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 16:46:23.454170 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446431    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 16:46:23.454170 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446434    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 16:46:23.454170 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446437    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 16:46:23.454170 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446441    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 16:46:23.454170 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446444    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 16:46:23.454170 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446448    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 16:46:23.454170 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446451    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 16:46:23.454170 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446454    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 16:46:23.454170 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446456    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 16:46:23.454170 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446459    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 16:46:23.454170 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446462    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 16:46:23.454170 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446464    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 16:46:23.454170 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446467    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 16:46:23.454170 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446469    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 16:46:23.454749 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446472    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 16:46:23.454749 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446475    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 16:46:23.454749 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446477    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 16:46:23.454749 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446480    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 16:46:23.454749 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446482    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 16:46:23.454749 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446490    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 16:46:23.454749 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446493    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 16:46:23.454749 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446496    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 16:46:23.454749 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446499    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 16:46:23.454749 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446501    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 16:46:23.454749 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446504    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 16:46:23.454749 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446506    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 16:46:23.454749 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446509    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 16:46:23.454749 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446512    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 16:46:23.454749 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446514    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 16:46:23.454749 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446517    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 16:46:23.454749 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446520    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 16:46:23.454749 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446523    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 16:46:23.454749 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446526    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 16:46:23.454749 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446528    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 16:46:23.455231 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446531    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 16:46:23.455231 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446534    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 16:46:23.455231 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446536    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 16:46:23.455231 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446539    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 16:46:23.455231 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446541    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 16:46:23.455231 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446544    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 16:46:23.455231 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446547    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 16:46:23.455231 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446549    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 16:46:23.455231 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446552    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 16:46:23.455231 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446555    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 16:46:23.455231 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446557    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 16:46:23.455231 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446560    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 16:46:23.455231 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446562    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 16:46:23.455231 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446565    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 16:46:23.455231 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446567    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 16:46:23.455231 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446570    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 16:46:23.455231 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446572    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 16:46:23.455231 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446575    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 16:46:23.455231 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446579    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 16:46:23.455708 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446581    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 16:46:23.455708 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446584    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 16:46:23.455708 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446587    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 16:46:23.455708 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446589    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 16:46:23.455708 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446592    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 16:46:23.455708 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446596    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 16:46:23.455708 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446600    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 16:46:23.455708 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446603    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 16:46:23.455708 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446606    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 16:46:23.455708 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446609    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 16:46:23.455708 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446611    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 16:46:23.455708 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446614    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 16:46:23.455708 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446616    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 16:46:23.455708 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446619    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 16:46:23.455708 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446621    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 16:46:23.455708 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446624    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 16:46:23.455708 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446627    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 16:46:23.455708 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446629    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 16:46:23.455708 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446632    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 16:46:23.456160 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446635    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 16:46:23.456160 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446637    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 16:46:23.456160 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446640    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 16:46:23.456160 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446642    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 16:46:23.456160 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446645    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 16:46:23.456160 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446647    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 16:46:23.456160 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446650    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 16:46:23.456160 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446652    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 16:46:23.456160 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446655    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 16:46:23.456160 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446657    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 16:46:23.456160 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446660    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 16:46:23.456160 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446662    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 16:46:23.456160 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446665    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 16:46:23.456160 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446667    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 16:46:23.456160 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446670    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 16:46:23.456160 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.446696    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 16:46:23.456566 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446729    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 16:46:23.456566 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446734    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 16:46:23.456566 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446737    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 16:46:23.456566 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446740    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 16:46:23.456566 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446744    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 16:46:23.456566 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446746    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 16:46:23.456566 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446749    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 16:46:23.456566 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446752    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 16:46:23.456566 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446755    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 16:46:23.456566 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446757    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 16:46:23.456566 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446760    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 16:46:23.456566 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446763    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 16:46:23.456566 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446765    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 16:46:23.456566 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446768    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 16:46:23.456566 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446770    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 16:46:23.456566 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446773    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 16:46:23.456566 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446775    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 16:46:23.456566 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446778    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 16:46:23.456566 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446780    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 16:46:23.456566 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446783    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 16:46:23.457018 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446785    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 16:46:23.457018 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446788    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 16:46:23.457018 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446790    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 16:46:23.457018 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446793    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 16:46:23.457018 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446796    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 16:46:23.457018 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446798    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 16:46:23.457018 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446801    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 16:46:23.457018 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446804    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 16:46:23.457018 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446806    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 16:46:23.457018 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446809    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 16:46:23.457018 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446813    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 16:46:23.457018 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446816    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 16:46:23.457018 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446818    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 16:46:23.457018 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446821    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 16:46:23.457018 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446823    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 16:46:23.457018 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446826    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 16:46:23.457018 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446829    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 16:46:23.457018 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446831    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 16:46:23.457018 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446834    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 16:46:23.457478 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446836    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 16:46:23.457478 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446840    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 16:46:23.457478 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446844    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 16:46:23.457478 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446847    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 16:46:23.457478 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446849    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 16:46:23.457478 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446852    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 16:46:23.457478 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446854    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 16:46:23.457478 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446857    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 16:46:23.457478 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446859    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 16:46:23.457478 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446862    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 16:46:23.457478 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446864    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 16:46:23.457478 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446867    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 16:46:23.457478 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446870    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 16:46:23.457478 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446872    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 16:46:23.457478 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446875    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 16:46:23.457478 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446877    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 16:46:23.457478 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446880    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 16:46:23.457478 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446882    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 16:46:23.457478 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446885    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 16:46:23.458025 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446887    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 16:46:23.458025 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446890    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 16:46:23.458025 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446893    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 16:46:23.458025 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446898    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 16:46:23.458025 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446901    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 16:46:23.458025 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446907    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 16:46:23.458025 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446910    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 16:46:23.458025 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446913    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 16:46:23.458025 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446916    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 16:46:23.458025 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446918    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 16:46:23.458025 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446921    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 16:46:23.458025 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446923    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 16:46:23.458025 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446926    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 16:46:23.458025 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446928    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 16:46:23.458025 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446931    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 16:46:23.458025 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446933    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 16:46:23.458025 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446936    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 16:46:23.458025 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446938    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 16:46:23.458025 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446941    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 16:46:23.458025 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446943    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 16:46:23.458635 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446946    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 16:46:23.458635 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446948    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 16:46:23.458635 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446951    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 16:46:23.458635 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446954    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 16:46:23.458635 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446957    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 16:46:23.458635 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446959    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 16:46:23.458635 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446962    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 16:46:23.458635 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446964    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 16:46:23.458635 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446967    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 16:46:23.458635 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446970    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 16:46:23.458635 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446973    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 16:46:23.458635 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446976    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 16:46:23.458635 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446979    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 16:46:23.458635 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446982    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 16:46:23.458635 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446985    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 16:46:23.458635 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446992    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 16:46:23.458635 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.446994    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 16:46:23.458635 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.447001    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 16:46:23.458635 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.447006    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 16:46:23.459147 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.447009    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 16:46:23.459147 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.447012    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 16:46:23.459147 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.447015    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 16:46:23.459147 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.447018    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 16:46:23.459147 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.447021    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 16:46:23.459147 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.447024    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 16:46:23.459147 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.447026    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 16:46:23.459147 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.447029    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 16:46:23.459147 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:23.447031    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 16:46:23.459147 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.447095    2581 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 16:46:23.459147 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.449083    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 16:46:23.459147 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.449800    2581 server.go:1014] "Starting client certificate rotation"
Oct 30 16:46:23.459147 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.449892    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 16:46:23.459147 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.449919    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 16:46:23.459147 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.451576    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 16:46:23.459147 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.453946    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 16:46:23.462587 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.462571    2581 log.go:25] "Validated CRI v1 runtime API"
Oct 30 16:46:23.465100 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.465085    2581 log.go:25] "Validated CRI v1 image API"
Oct 30 16:46:23.465949 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.465920    2581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 16:46:23.469622 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.469602    2581 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 dc6bcaed-293a-4f46-8c6d-e7787d9ac2a4:/dev/nvme0n1p3 de84304a-56b4-4fe4-82a0-36101d1a21a1:/dev/nvme0n1p4]
Oct 30 16:46:23.469686 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.469625    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 16:46:23.475504 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.475368    2581 manager.go:217] Machine: {Timestamp:2026-10-30 16:46:23.474890848 +0000 UTC m=+0.257206102 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3092764 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ae7ffdff0fde93c8907da133c979c SystemUUID:ec2ae7ff-dff0-fde9-3c89-07da133c979c BootID:f25f2461-6b04-4036-94db-ce6afe849079 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ea:b9:8d:51 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ea:b9:8d:51 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d6:2a:94:49:57:e2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 16:46:23.475504 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.475494    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 16:46:23.475624 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.475581    2581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 16:46:23.476601 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.476575    2581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 16:46:23.476736 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.476603    2581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-13.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 16:46:23.476777 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.476739    2581 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 16:46:23.476777 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.476747    2581 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 16:46:23.476777 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.476758    2581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 16:46:23.476777 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.476770    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 16:46:23.477519 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.477509    2581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 16:46:23.481612 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.481600    2581 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 16:46:23.481728 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.481718    2581 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 16:46:23.482228 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.482219    2581 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 16:46:23.482259 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.482232    2581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 16:46:23.482259 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.482242    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 16:46:23.482259 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.482251    2581 kubelet.go:405] "Adding apiserver pod source"
Oct 30 16:46:23.482342 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.482260    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 16:46:23.482672 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.482659    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 16:46:23.484467 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.484440    2581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 16:46:23.488299 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.488286    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 16:46:23.488975 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.488961    2581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 16:46:23.489027 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.488994    2581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 16:46:23.489181 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.489171    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 16:46:23.489213 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.489191    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 16:46:23.489213 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.489205    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 16:46:23.489269 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.489219    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 16:46:23.489269 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.489234    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 16:46:23.489269 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.489243    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 16:46:23.489681 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.489673    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 16:46:23.489717 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.489682    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 16:46:23.489717 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.489697    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 16:46:23.489717 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.489703    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 16:46:23.489717 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.489718    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 16:46:23.489842 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.489727    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 16:46:23.490221 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.490213    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 16:46:23.490221 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.490221    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 16:46:23.494036 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.494022    2581 server.go:1263] "Started kubelet"
Oct 30 16:46:23.494170 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.494137    2581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 16:46:23.494956 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.494253    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 16:46:23.495034 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.494977    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 16:46:23.495258 ip-10-0-131-13 systemd[1]: Started Kubernetes Kubelet.
Oct 30 16:46:23.496018 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.495997    2581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 16:46:23.496974 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.496955    2581 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 16:46:23.500147 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:23.500124    2581 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 16:46:23.501434 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.501399    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 16:46:23.501763 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.501704    2581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 16:46:23.501988 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:23.501944    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-13.ec2.internal\" not found"
Oct 30 16:46:23.502707 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.502329    2581 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 16:46:23.502707 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.502346    2581 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 16:46:23.502707 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.502329    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 16:46:23.502707 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.502436    2581 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 16:46:23.502707 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.502443    2581 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 16:46:23.503174 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.503161    2581 factory.go:153] Registering CRI-O factory
Oct 30 16:46:23.503213 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.503177    2581 factory.go:223] Registration of the crio container factory successfully
Oct 30 16:46:23.503241 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.503225    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 16:46:23.503241 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.503232    2581 factory.go:55] Registering systemd factory
Oct 30 16:46:23.503241 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.503238    2581 factory.go:223] Registration of the systemd container factory successfully
Oct 30 16:46:23.503327 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.503250    2581 factory.go:103] Registering Raw factory
Oct 30 16:46:23.503327 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.503274    2581 manager.go:1199] Started watching for new ooms in manager
Oct 30 16:46:23.504157 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.504138    2581 manager.go:319] Starting recovery of all containers
Oct 30 16:46:23.514201 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.514175    2581 manager.go:324] Recovery completed
Oct 30 16:46:23.520265 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.520225    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 16:46:23.522300 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.522279    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-13.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 16:46:23.522416 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.522315    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-13.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 16:46:23.522416 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.522331    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-13.ec2.internal" event="NodeHasSufficientPID"
Oct 30 16:46:23.522994 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.522977    2581 cpu_manager.go:226] "Starting" policy="none"
Oct 30 16:46:23.522994 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.522992    2581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 16:46:23.523126 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.523015    2581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 16:46:23.523126 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.523040    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 16:46:23.523941 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.523926    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-13.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 16:46:23.524024 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.523951    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-13.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 16:46:23.524024 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.523961    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-13.ec2.internal" event="NodeHasSufficientPID"
Oct 30 16:46:23.525341 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.525329    2581 policy_none.go:50] "Start"
Oct 30 16:46:23.525393 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.525345    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 16:46:23.525393 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.525355    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 16:46:23.527341 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.527330    2581 policy_none.go:44] "Start"
Oct 30 16:46:23.535281 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.535260    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-13.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 16:46:23.535528 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:23.535501    2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-13.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 16:46:23.536146 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:23.535267    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-13.ec2.internal.18e35c8079e1808f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-13.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-13.ec2.internal,},FirstTimestamp:2026-10-30 16:46:23.493980303 +0000 UTC m=+0.276295558,LastTimestamp:2026-10-30 16:46:23.493980303 +0000 UTC m=+0.276295558,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-13.ec2.internal,}"
Oct 30 16:46:23.543346 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.543311    2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 16:46:23.550733 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:23.550705    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 16:46:23.550880 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:23.550855    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-13.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 16:46:23.550956 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:23.550872    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-13.ec2.internal.18e35c807b919d3d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-13.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-13.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-13.ec2.internal,},FirstTimestamp:2026-10-30 16:46:23.522299197 +0000 UTC m=+0.304614462,LastTimestamp:2026-10-30 16:46:23.522299197 +0000 UTC m=+0.304614462,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-13.ec2.internal,}"
Oct 30 16:46:23.557923 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.557894    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6mb7f"
Oct 30 16:46:23.559104 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.559074    2581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 16:46:23.563361 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.563345    2581 manager.go:341] "Starting Device Plugin manager"
Oct 30 16:46:23.563481 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:23.563381    2581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 16:46:23.563481 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.563394    2581 server.go:87] "Starting device plugin registration server"
Oct 30 16:46:23.563742 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.563730    2581 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 16:46:23.563772 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.563745    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 16:46:23.564018 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.563843    2581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 16:46:23.564018 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.563951    2581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 16:46:23.564018 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.563960    2581 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 16:46:23.564591 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:23.564565    2581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 16:46:23.564686 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:23.564617    2581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-13.ec2.internal\" not found"
Oct 30 16:46:23.568884 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.568864    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6mb7f"
Oct 30 16:46:23.597794 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:23.581447    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-13.ec2.internal.18e35c807b91fa77  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-13.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-13.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-13.ec2.internal,},FirstTimestamp:2026-10-30 16:46:23.522323063 +0000 UTC m=+0.304638323,LastTimestamp:2026-10-30 16:46:23.522323063 +0000 UTC m=+0.304638323,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-13.ec2.internal,}"
Oct 30 16:46:23.597794 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:23.581572    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 16:46:23.650852 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.650798    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 16:46:23.652056 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.652039    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 16:46:23.652056 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.652061    2581 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 16:46:23.652192 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.652083    2581 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 16:46:23.652192 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:23.652120    2581 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 16:46:23.664807 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.664785    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 16:46:23.666426 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.666390    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-13.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 16:46:23.666544 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.666444    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-13.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 16:46:23.666544 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.666459    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-13.ec2.internal" event="NodeHasSufficientPID"
Oct 30 16:46:23.666544 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.666490    2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-13.ec2.internal"
Oct 30 16:46:23.674546 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.674530    2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 16:46:23.693276 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.693213    2581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-13.ec2.internal"
Oct 30 16:46:23.693276 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:23.693242    2581 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-13.ec2.internal\": node \"ip-10-0-131-13.ec2.internal\" not found"
Oct 30 16:46:23.740613 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:23.740582    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-13.ec2.internal\" not found"
Oct 30 16:46:23.753263 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.753232    2581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-13.ec2.internal"]
Oct 30 16:46:23.753382 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.753306    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 16:46:23.756492 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.756473    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-13.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 16:46:23.756587 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.756508    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-13.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 16:46:23.756587 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.756518    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-13.ec2.internal" event="NodeHasSufficientPID"
Oct 30 16:46:23.758882 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.758870    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 16:46:23.759025 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.759011    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal"
Oct 30 16:46:23.759061 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.759042    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 16:46:23.763651 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.763634    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-13.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 16:46:23.763763 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.763668    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-13.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 16:46:23.763763 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.763684    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-13.ec2.internal" event="NodeHasSufficientPID"
Oct 30 16:46:23.763763 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.763633    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-13.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 16:46:23.763763 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.763740    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-13.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 16:46:23.763763 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.763752    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-13.ec2.internal" event="NodeHasSufficientPID"
Oct 30 16:46:23.765940 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.765925    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-13.ec2.internal"
Oct 30 16:46:23.766006 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.765951    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 16:46:23.766688 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.766670    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-13.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 16:46:23.766757 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.766696    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-13.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 16:46:23.766757 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.766708    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-13.ec2.internal" event="NodeHasSufficientPID"
Oct 30 16:46:23.801342 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:23.801320    2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-13.ec2.internal\" not found" node="ip-10-0-131-13.ec2.internal"
Oct 30 16:46:23.803887 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:23.803868    2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-13.ec2.internal\" not found" node="ip-10-0-131-13.ec2.internal"
Oct 30 16:46:23.803989 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.803971    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91ca0575ad91df65b05568b935060571-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal\" (UID: \"91ca0575ad91df65b05568b935060571\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal"
Oct 30 16:46:23.804038 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.803997    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91ca0575ad91df65b05568b935060571-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal\" (UID: \"91ca0575ad91df65b05568b935060571\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal"
Oct 30 16:46:23.804038 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.804012    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/777e8481b0741b0fafcaf01b060f8274-config\") pod \"kube-apiserver-proxy-ip-10-0-131-13.ec2.internal\" (UID: \"777e8481b0741b0fafcaf01b060f8274\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-13.ec2.internal"
Oct 30 16:46:23.841439 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:23.841388    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-13.ec2.internal\" not found"
Oct 30 16:46:23.905128 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.905102    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91ca0575ad91df65b05568b935060571-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal\" (UID: \"91ca0575ad91df65b05568b935060571\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal"
Oct 30 16:46:23.905206 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.905134    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/777e8481b0741b0fafcaf01b060f8274-config\") pod \"kube-apiserver-proxy-ip-10-0-131-13.ec2.internal\" (UID: \"777e8481b0741b0fafcaf01b060f8274\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-13.ec2.internal"
Oct 30 16:46:23.905206 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.905152    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91ca0575ad91df65b05568b935060571-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal\" (UID: \"91ca0575ad91df65b05568b935060571\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal"
Oct 30 16:46:23.905206 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.905199    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91ca0575ad91df65b05568b935060571-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal\" (UID: \"91ca0575ad91df65b05568b935060571\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal"
Oct 30 16:46:23.905319 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.905203    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91ca0575ad91df65b05568b935060571-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal\" (UID: \"91ca0575ad91df65b05568b935060571\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal"
Oct 30 16:46:23.905319 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:23.905215    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/777e8481b0741b0fafcaf01b060f8274-config\") pod \"kube-apiserver-proxy-ip-10-0-131-13.ec2.internal\" (UID: \"777e8481b0741b0fafcaf01b060f8274\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-13.ec2.internal"
Oct 30 16:46:23.942242 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:23.942208    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-13.ec2.internal\" not found"
Oct 30 16:46:24.042590 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:24.042511    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-13.ec2.internal\" not found"
Oct 30 16:46:24.103718 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:24.103693    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal"
Oct 30 16:46:24.106877 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:24.106856    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-13.ec2.internal"
Oct 30 16:46:24.143097 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:24.143062    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-13.ec2.internal\" not found"
Oct 30 16:46:24.243650 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:24.243618    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-13.ec2.internal\" not found"
Oct 30 16:46:24.344185 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:24.344105    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-13.ec2.internal\" not found"
Oct 30 16:46:24.445277 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:24.445230    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-13.ec2.internal\" not found"
Oct 30 16:46:24.449680 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:24.449654    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 16:46:24.449863 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:24.449841    2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 16:46:24.546373 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:24.546342    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-13.ec2.internal\" not found"
Oct 30 16:46:24.558184 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:24.558155    2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 16:46:24.573986 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:24.573941    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 16:41:24 +0000 UTC" deadline="2028-08-11 10:57:17.81021505 +0000 UTC"
Oct 30 16:46:24.573986 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:24.573985    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15618h10m53.236235897s"
Oct 30 16:46:24.647519 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:24.647494    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-13.ec2.internal\" not found"
Oct 30 16:46:24.649526 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:24.649495    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ca0575ad91df65b05568b935060571.slice/crio-6b186c5c69966ccc90f463d36916ec22aa217e88fdd7539d03eac4722944334f WatchSource:0}: Error finding container 6b186c5c69966ccc90f463d36916ec22aa217e88fdd7539d03eac4722944334f: Status 404 returned error can't find the container with id 6b186c5c69966ccc90f463d36916ec22aa217e88fdd7539d03eac4722944334f
Oct 30 16:46:24.649916 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:24.649895    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod777e8481b0741b0fafcaf01b060f8274.slice/crio-a987ba2e91dc2167bc9c214225be559a90d87b8b84f9ab76da87df360c9f4d3e WatchSource:0}: Error finding container a987ba2e91dc2167bc9c214225be559a90d87b8b84f9ab76da87df360c9f4d3e: Status 404 returned error can't find the container with id a987ba2e91dc2167bc9c214225be559a90d87b8b84f9ab76da87df360c9f4d3e
Oct 30 16:46:24.654608 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:24.654586    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 16:46:24.655504 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:24.655462    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal" event={"ID":"91ca0575ad91df65b05568b935060571","Type":"ContainerStarted","Data":"6b186c5c69966ccc90f463d36916ec22aa217e88fdd7539d03eac4722944334f"}
Oct 30 16:46:24.656637 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:24.656611    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-13.ec2.internal" event={"ID":"777e8481b0741b0fafcaf01b060f8274","Type":"ContainerStarted","Data":"a987ba2e91dc2167bc9c214225be559a90d87b8b84f9ab76da87df360c9f4d3e"}
Oct 30 16:46:24.741880 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:24.741835    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 16:46:24.743262 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:24.743240    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-13.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 16:46:24.743376 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:24.743273    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-13.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 16:46:24.743376 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:24.743285    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-13.ec2.internal" event="NodeHasSufficientPID"
Oct 30 16:46:24.748564 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:24.748540    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-13.ec2.internal\" not found"
Oct 30 16:46:24.848671 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:24.848632    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-13.ec2.internal\" not found"
Oct 30 16:46:24.865569 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:24.865542    2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 16:46:24.921864 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:24.921789    2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 16:46:25.003270 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.003231    2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal"
Oct 30 16:46:25.032448 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.032428    2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 16:46:25.033393 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.033378    2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-13.ec2.internal"
Oct 30 16:46:25.058973 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.058942    2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 16:46:25.277251 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.277197    2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 16:46:25.483418 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.483359    2581 apiserver.go:52] "Watching apiserver"
Oct 30 16:46:25.503292 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.502670    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 16:46:25.519984 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.519948    2581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 16:46:25.520338 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.520314    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-26t96","kube-system/konnectivity-agent-vw5v7","kube-system/kube-apiserver-proxy-ip-10-0-131-13.ec2.internal","openshift-image-registry/node-ca-jhqzq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal","openshift-multus/multus-8t9lr","kube-system/global-pull-secret-syncer-5cgn8","openshift-dns/node-resolver-nlr68","openshift-ovn-kubernetes/ovnkube-node-kkb8g","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf","openshift-cluster-node-tuning-operator/tuned-f8wv9","openshift-multus/multus-additional-cni-plugins-bgwcr","openshift-multus/network-metrics-daemon-7zf5f","openshift-network-diagnostics/network-check-target-r2pxh"]
Oct 30 16:46:25.523037 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.523008    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:46:25.523159 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:25.523104    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cgn8" podUID="498ef897-364c-4e19-9fbf-cdc46aac623f"
Oct 30 16:46:25.525486 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.525456    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.527764 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.527704    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2pxh"
Oct 30 16:46:25.527871 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:25.527786    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2pxh" podUID="7bfc195d-6fbc-45c8-9e2a-f9c206838651"
Oct 30 16:46:25.530106 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.530080    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jhqzq"
Oct 30 16:46:25.532774 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.532430    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nlr68"
Oct 30 16:46:25.534805 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.534773    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.537423 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.537398    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.540229 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.540207    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:46:25.540333 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:25.540302    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zf5f" podUID="980827e6-3d1b-4924-acde-8d408d5e50df"
Oct 30 16:46:25.542596 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.542573    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vw5v7"
Oct 30 16:46:25.545114 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.545092    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:46:25.547828 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.547805    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-26t96"
Oct 30 16:46:25.548383 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.548364    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 16:46:25.548383 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.548359    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rbwcw\""
Oct 30 16:46:25.548563 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.548378    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 16:46:25.549219 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.549203    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 16:46:25.549781 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.549765    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fxkbf\""
Oct 30 16:46:25.550187 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.550169    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 16:46:25.550269 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.550255    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 16:46:25.551015 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.550995    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:46:25.556975 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.556944    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 16:46:25.557074 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.557023    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 16:46:25.557177 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.557161    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 16:46:25.557217 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.557165    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 16:46:25.557541 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.557525    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 16:46:25.561368 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.558629    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 16:46:25.561368 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.558666    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6bljg\""
Oct 30 16:46:25.561368 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.558694    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 16:46:25.561368 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.558785    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 16:46:25.561368 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.558992    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wd4nn\""
Oct 30 16:46:25.561368 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.559591    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pdblj\""
Oct 30 16:46:25.561368 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.559659    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 16:46:25.561368 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.559717    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 16:46:25.561368 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.559770    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 16:46:25.561368 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.559954    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 16:46:25.564004 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.563984    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 16:46:25.564124 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.564104    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 16:46:25.564240 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.564187    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-z7xnd\""
Oct 30 16:46:25.582441 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.582410    2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 16:46:25.582686 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.582502    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bbcvc"
Oct 30 16:46:25.596189 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.596163    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bbcvc"
Oct 30 16:46:25.603343 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.603317    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 16:46:25.615545 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.615512    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/498ef897-364c-4e19-9fbf-cdc46aac623f-kubelet-config\") pod \"global-pull-secret-syncer-5cgn8\" (UID: \"498ef897-364c-4e19-9fbf-cdc46aac623f\") " pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:46:25.615545 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.615551    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-system-cni-dir\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.615774 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.615568    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-multus-cni-dir\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.615774 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.615590    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-registration-dir\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:46:25.615774 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.615657    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-var-lib-kubelet\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.615774 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.615713    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-run-ovn-kubernetes\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.615940 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.615784    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-sys-fs\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:46:25.615940 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.615829    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdd6c\" (UniqueName: \"kubernetes.io/projected/cf8a28a6-fd11-45dc-9927-adeba7780b46-kube-api-access-vdd6c\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.615940 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.615862    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5wnk\" (UniqueName: \"kubernetes.io/projected/92b17f34-17e7-4d4f-b43a-4e828ede0187-kube-api-access-r5wnk\") pod \"node-resolver-nlr68\" (UID: \"92b17f34-17e7-4d4f-b43a-4e828ede0187\") " pod="openshift-dns/node-resolver-nlr68"
Oct 30 16:46:25.615940 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.615911    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6c03c304-7756-45bb-9597-350026ed361e-konnectivity-ca\") pod \"konnectivity-agent-vw5v7\" (UID: \"6c03c304-7756-45bb-9597-350026ed361e\") " pod="kube-system/konnectivity-agent-vw5v7"
Oct 30 16:46:25.616075 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.615955    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkqgx\" (UniqueName: \"kubernetes.io/projected/db037093-50f6-49b9-93c2-f37a7f1290d6-kube-api-access-mkqgx\") pod \"node-ca-jhqzq\" (UID: \"db037093-50f6-49b9-93c2-f37a7f1290d6\") " pod="openshift-image-registry/node-ca-jhqzq"
Oct 30 16:46:25.616075 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.615979    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-run-openvswitch\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.616075 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.616003    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60f548f1-6eae-4929-bfd7-db1db9aab72e-cni-binary-copy\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:46:25.616075 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.616062    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/60f548f1-6eae-4929-bfd7-db1db9aab72e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:46:25.616229 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.616122    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/498ef897-364c-4e19-9fbf-cdc46aac623f-dbus\") pod \"global-pull-secret-syncer-5cgn8\" (UID: \"498ef897-364c-4e19-9fbf-cdc46aac623f\") " pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:46:25.616229 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.616168    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-cnibin\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.616229 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.616188    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60154025-ad1c-41b0-b28c-1a83e5994685-cni-binary-copy\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.616229 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.616205    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-multus-conf-dir\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.616379 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.616243    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/33a051e9-adff-4247-8988-f6ea635aaee0-iptables-alerter-script\") pod \"iptables-alerter-26t96\" (UID: \"33a051e9-adff-4247-8988-f6ea635aaee0\") " pod="openshift-network-operator/iptables-alerter-26t96"
Oct 30 16:46:25.616379 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.616262    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92b17f34-17e7-4d4f-b43a-4e828ede0187-tmp-dir\") pod \"node-resolver-nlr68\" (UID: \"92b17f34-17e7-4d4f-b43a-4e828ede0187\") " pod="openshift-dns/node-resolver-nlr68"
Oct 30 16:46:25.616379 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.616281    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60f548f1-6eae-4929-bfd7-db1db9aab72e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:46:25.616379 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.616307    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs\") pod \"network-metrics-daemon-7zf5f\" (UID: \"980827e6-3d1b-4924-acde-8d408d5e50df\") " pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:46:25.616379 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.616349    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-modprobe-d\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.616578 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.616397    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-run-ovn\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.616578 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.616465    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.616578 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.616509    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d90edf93-884d-46e3-b1f7-dcae29944ca2-ovnkube-script-lib\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.616691 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.616578    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-etc-kubernetes\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.616691 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.616604    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-socket-dir\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:46:25.616691 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.616635    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33a051e9-adff-4247-8988-f6ea635aaee0-host-slash\") pod \"iptables-alerter-26t96\" (UID: \"33a051e9-adff-4247-8988-f6ea635aaee0\") " pod="openshift-network-operator/iptables-alerter-26t96"
Oct 30 16:46:25.616691 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.616665    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-kubelet\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.616691 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.616686    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-cni-netd\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.616887 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.616710    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d90edf93-884d-46e3-b1f7-dcae29944ca2-ovn-node-metrics-cert\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.616887 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.616730    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-run\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.616887 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.616761    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-sys\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.616887 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.616792    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-run-systemd\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.616887 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.616811    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60f548f1-6eae-4929-bfd7-db1db9aab72e-system-cni-dir\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:46:25.616887 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.616839    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/498ef897-364c-4e19-9fbf-cdc46aac623f-original-pull-secret\") pod \"global-pull-secret-syncer-5cgn8\" (UID: \"498ef897-364c-4e19-9fbf-cdc46aac623f\") " pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:46:25.616887 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.616859    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-hostroot\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.617115 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.616896    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-kubernetes\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.617115 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.616912    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-sysctl-conf\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.617115 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.616929    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-host\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.617115 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.616975    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf8a28a6-fd11-45dc-9927-adeba7780b46-tmp\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.617115 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.616992    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-systemd\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.617115 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617008    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-tuned\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.617115 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617024    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxv5t\" (UniqueName: \"kubernetes.io/projected/d90edf93-884d-46e3-b1f7-dcae29944ca2-kube-api-access-cxv5t\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.617115 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617040    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60f548f1-6eae-4929-bfd7-db1db9aab72e-cnibin\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:46:25.617115 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617064    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/60f548f1-6eae-4929-bfd7-db1db9aab72e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:46:25.617115 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617083    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-var-lib-cni-bin\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.617115 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617098    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-var-lib-kubelet\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.617549 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617114    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/60154025-ad1c-41b0-b28c-1a83e5994685-multus-daemon-config\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.617549 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617153    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5l7w\" (UniqueName: \"kubernetes.io/projected/60154025-ad1c-41b0-b28c-1a83e5994685-kube-api-access-f5l7w\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.617549 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617169    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jch5v\" (UniqueName: \"kubernetes.io/projected/980827e6-3d1b-4924-acde-8d408d5e50df-kube-api-access-jch5v\") pod \"network-metrics-daemon-7zf5f\" (UID: \"980827e6-3d1b-4924-acde-8d408d5e50df\") " pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:46:25.617549 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617186    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgh6f\" (UniqueName: \"kubernetes.io/projected/5cf6407b-d0de-4fa7-a346-1f211ab4c792-kube-api-access-wgh6f\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:46:25.617549 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617204    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-sysconfig\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.617549 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617229    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-var-lib-openvswitch\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.617549 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617296    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-var-lib-cni-multus\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.617549 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617322    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/db037093-50f6-49b9-93c2-f37a7f1290d6-serviceca\") pod \"node-ca-jhqzq\" (UID: \"db037093-50f6-49b9-93c2-f37a7f1290d6\") " pod="openshift-image-registry/node-ca-jhqzq"
Oct 30 16:46:25.617549 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617347    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-etc-openvswitch\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.617549 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617363    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-log-socket\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.617549 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617378    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-sysctl-d\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.617549 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617393    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/92b17f34-17e7-4d4f-b43a-4e828ede0187-hosts-file\") pod \"node-resolver-nlr68\" (UID: \"92b17f34-17e7-4d4f-b43a-4e828ede0187\") " pod="openshift-dns/node-resolver-nlr68"
Oct 30 16:46:25.617549 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617408    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-slash\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.617549 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617426    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhxl7\" (UniqueName: \"kubernetes.io/projected/7bfc195d-6fbc-45c8-9e2a-f9c206838651-kube-api-access-jhxl7\") pod \"network-check-target-r2pxh\" (UID: \"7bfc195d-6fbc-45c8-9e2a-f9c206838651\") " pod="openshift-network-diagnostics/network-check-target-r2pxh"
Oct 30 16:46:25.617549 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617446    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:46:25.617549 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617465    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-etc-selinux\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:46:25.618069 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617510    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-run-netns\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.618069 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617532    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-cni-bin\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.618069 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617552    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6jfp\" (UniqueName: \"kubernetes.io/projected/60f548f1-6eae-4929-bfd7-db1db9aab72e-kube-api-access-k6jfp\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:46:25.618069 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617573    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-run-netns\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.618069 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617614    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-device-dir\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:46:25.618069 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617636    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d90edf93-884d-46e3-b1f7-dcae29944ca2-ovnkube-config\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.618069 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617656    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-run-k8s-cni-cncf-io\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.618069 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617676    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-multus-socket-dir-parent\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.618069 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617700    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-run-multus-certs\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.618069 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617717    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x97hn\" (UniqueName: \"kubernetes.io/projected/33a051e9-adff-4247-8988-f6ea635aaee0-kube-api-access-x97hn\") pod \"iptables-alerter-26t96\" (UID: \"33a051e9-adff-4247-8988-f6ea635aaee0\") " pod="openshift-network-operator/iptables-alerter-26t96"
Oct 30 16:46:25.618069 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617739    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-lib-modules\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.618069 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617754    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-node-log\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.618069 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617773    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d90edf93-884d-46e3-b1f7-dcae29944ca2-env-overrides\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.618069 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617793    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6c03c304-7756-45bb-9597-350026ed361e-agent-certs\") pod \"konnectivity-agent-vw5v7\" (UID: \"6c03c304-7756-45bb-9597-350026ed361e\") " pod="kube-system/konnectivity-agent-vw5v7"
Oct 30 16:46:25.618069 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617808    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db037093-50f6-49b9-93c2-f37a7f1290d6-host\") pod \"node-ca-jhqzq\" (UID: \"db037093-50f6-49b9-93c2-f37a7f1290d6\") " pod="openshift-image-registry/node-ca-jhqzq"
Oct 30 16:46:25.618069 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.617823    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-systemd-units\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.657203 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.657168    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-j4x52\""
Oct 30 16:46:25.657393 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.657219    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 16:46:25.657393 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.657270    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f7c68\""
Oct 30 16:46:25.657393 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.657363    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8fhb8\""
Oct 30 16:46:25.657641 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.657612    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 16:46:25.657696 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.657647    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 16:46:25.657750 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.657731    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 16:46:25.657803 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.657788    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 16:46:25.657803 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.657796    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 16:46:25.657904 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.657861    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 16:46:25.658046 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.658023    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 16:46:25.718218 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.718175    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkqgx\" (UniqueName: \"kubernetes.io/projected/db037093-50f6-49b9-93c2-f37a7f1290d6-kube-api-access-mkqgx\") pod \"node-ca-jhqzq\" (UID: \"db037093-50f6-49b9-93c2-f37a7f1290d6\") " pod="openshift-image-registry/node-ca-jhqzq"
Oct 30 16:46:25.718218 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.718211    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-run-openvswitch\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.718488 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.718240    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60f548f1-6eae-4929-bfd7-db1db9aab72e-cni-binary-copy\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:46:25.718488 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.718266    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/60f548f1-6eae-4929-bfd7-db1db9aab72e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:46:25.718488 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.718293    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/498ef897-364c-4e19-9fbf-cdc46aac623f-dbus\") pod \"global-pull-secret-syncer-5cgn8\" (UID: \"498ef897-364c-4e19-9fbf-cdc46aac623f\") " pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:46:25.718488 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.718318    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-cnibin\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.718488 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.718330    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-run-openvswitch\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.718753 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.718615    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-cnibin\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.718753 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.718343    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60154025-ad1c-41b0-b28c-1a83e5994685-cni-binary-copy\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.718753 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.718676    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-multus-conf-dir\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.718753 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.718685    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/498ef897-364c-4e19-9fbf-cdc46aac623f-dbus\") pod \"global-pull-secret-syncer-5cgn8\" (UID: \"498ef897-364c-4e19-9fbf-cdc46aac623f\") " pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:46:25.718753 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.718704    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/33a051e9-adff-4247-8988-f6ea635aaee0-iptables-alerter-script\") pod \"iptables-alerter-26t96\" (UID: \"33a051e9-adff-4247-8988-f6ea635aaee0\") " pod="openshift-network-operator/iptables-alerter-26t96"
Oct 30 16:46:25.719103 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719082    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60154025-ad1c-41b0-b28c-1a83e5994685-cni-binary-copy\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.719179 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719100    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60f548f1-6eae-4929-bfd7-db1db9aab72e-cni-binary-copy\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:46:25.719179 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719144    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92b17f34-17e7-4d4f-b43a-4e828ede0187-tmp-dir\") pod \"node-resolver-nlr68\" (UID: \"92b17f34-17e7-4d4f-b43a-4e828ede0187\") " pod="openshift-dns/node-resolver-nlr68"
Oct 30 16:46:25.719290 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719177    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60f548f1-6eae-4929-bfd7-db1db9aab72e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:46:25.719290 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719207    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs\") pod \"network-metrics-daemon-7zf5f\" (UID: \"980827e6-3d1b-4924-acde-8d408d5e50df\") " pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:46:25.719290 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719221    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/60f548f1-6eae-4929-bfd7-db1db9aab72e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:46:25.719290 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719229    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/33a051e9-adff-4247-8988-f6ea635aaee0-iptables-alerter-script\") pod \"iptables-alerter-26t96\" (UID: \"33a051e9-adff-4247-8988-f6ea635aaee0\") " pod="openshift-network-operator/iptables-alerter-26t96"
Oct 30 16:46:25.719290 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719234    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-modprobe-d\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.719290 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719272    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-multus-conf-dir\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.719290 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719281    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-run-ovn\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.719662 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719301    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.719662 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719319    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d90edf93-884d-46e3-b1f7-dcae29944ca2-ovnkube-script-lib\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.719662 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719334    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-etc-kubernetes\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.719662 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719334    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-modprobe-d\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.719662 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719356    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-socket-dir\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:46:25.719662 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719382    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33a051e9-adff-4247-8988-f6ea635aaee0-host-slash\") pod \"iptables-alerter-26t96\" (UID: \"33a051e9-adff-4247-8988-f6ea635aaee0\") " pod="openshift-network-operator/iptables-alerter-26t96"
Oct 30 16:46:25.719662 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719402    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-kubelet\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.719662 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719418    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-cni-netd\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.719662 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719424    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60f548f1-6eae-4929-bfd7-db1db9aab72e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:46:25.719662 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:25.719430    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 16:46:25.719662 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719432    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92b17f34-17e7-4d4f-b43a-4e828ede0187-tmp-dir\") pod \"node-resolver-nlr68\" (UID: \"92b17f34-17e7-4d4f-b43a-4e828ede0187\") " pod="openshift-dns/node-resolver-nlr68"
Oct 30 16:46:25.719662 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719475    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-run-ovn\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.719662 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719524    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-etc-kubernetes\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.719662 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:25.719538    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs podName:980827e6-3d1b-4924-acde-8d408d5e50df nodeName:}" failed. No retries permitted until 2026-10-30 16:46:26.219479586 +0000 UTC m=+3.001794849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs") pod "network-metrics-daemon-7zf5f" (UID: "980827e6-3d1b-4924-acde-8d408d5e50df") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 16:46:25.719662 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719545    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.719662 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719578    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33a051e9-adff-4247-8988-f6ea635aaee0-host-slash\") pod \"iptables-alerter-26t96\" (UID: \"33a051e9-adff-4247-8988-f6ea635aaee0\") " pod="openshift-network-operator/iptables-alerter-26t96"
Oct 30 16:46:25.720441 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719693    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-socket-dir\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:46:25.720441 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719696    2581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 16:46:25.720441 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719437    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d90edf93-884d-46e3-b1f7-dcae29944ca2-ovn-node-metrics-cert\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.720441 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719743    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-kubelet\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.720441 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719745    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-run\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.720441 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719781    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-sys\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.720441 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719795    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-run\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.720441 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719808    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-run-systemd\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.720441 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719833    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60f548f1-6eae-4929-bfd7-db1db9aab72e-system-cni-dir\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:46:25.720441 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719853    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-sys\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.720441 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719859    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/498ef897-364c-4e19-9fbf-cdc46aac623f-original-pull-secret\") pod \"global-pull-secret-syncer-5cgn8\" (UID: \"498ef897-364c-4e19-9fbf-cdc46aac623f\") " pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:46:25.720441 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719889    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-hostroot\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.720441 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719899    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-cni-netd\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.720441 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719933    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-hostroot\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.720441 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719934    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60f548f1-6eae-4929-bfd7-db1db9aab72e-system-cni-dir\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:46:25.720441 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719951    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-run-systemd\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.720441 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719951    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-kubernetes\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.720441 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719991    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-sysctl-conf\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.720441 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:25.720014    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 16:46:25.721422 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720021    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-host\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.721422 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720045    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf8a28a6-fd11-45dc-9927-adeba7780b46-tmp\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.721422 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:25.720063    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/498ef897-364c-4e19-9fbf-cdc46aac623f-original-pull-secret podName:498ef897-364c-4e19-9fbf-cdc46aac623f nodeName:}" failed. No retries permitted until 2026-10-30 16:46:26.220047054 +0000 UTC m=+3.002362312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/498ef897-364c-4e19-9fbf-cdc46aac623f-original-pull-secret") pod "global-pull-secret-syncer-5cgn8" (UID: "498ef897-364c-4e19-9fbf-cdc46aac623f") : object "kube-system"/"original-pull-secret" not registered
Oct 30 16:46:25.721422 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.719995    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-kubernetes\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.721422 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720082    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-sysctl-conf\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.721422 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720093    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-systemd\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.721422 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720111    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d90edf93-884d-46e3-b1f7-dcae29944ca2-ovnkube-script-lib\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.721422 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720122    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-tuned\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.721422 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720153    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxv5t\" (UniqueName: \"kubernetes.io/projected/d90edf93-884d-46e3-b1f7-dcae29944ca2-kube-api-access-cxv5t\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.721422 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720174    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-systemd\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.721422 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720179    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60f548f1-6eae-4929-bfd7-db1db9aab72e-cnibin\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:46:25.721422 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720124    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-host\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.721422 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720211    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/60f548f1-6eae-4929-bfd7-db1db9aab72e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:46:25.721422 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720258    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-var-lib-cni-bin\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.721422 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720290    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-var-lib-kubelet\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.721422 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/60154025-ad1c-41b0-b28c-1a83e5994685-multus-daemon-config\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.721422 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720336    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5l7w\" (UniqueName: \"kubernetes.io/projected/60154025-ad1c-41b0-b28c-1a83e5994685-kube-api-access-f5l7w\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.722301 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720360    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jch5v\" (UniqueName: \"kubernetes.io/projected/980827e6-3d1b-4924-acde-8d408d5e50df-kube-api-access-jch5v\") pod \"network-metrics-daemon-7zf5f\" (UID: \"980827e6-3d1b-4924-acde-8d408d5e50df\") " pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:46:25.722301 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720384    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgh6f\" (UniqueName: \"kubernetes.io/projected/5cf6407b-d0de-4fa7-a346-1f211ab4c792-kube-api-access-wgh6f\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:46:25.722301 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720405    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-sysconfig\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.722301 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720429    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-var-lib-openvswitch\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.722301 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720453    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-var-lib-cni-multus\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.722301 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720487    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/db037093-50f6-49b9-93c2-f37a7f1290d6-serviceca\") pod \"node-ca-jhqzq\" (UID: \"db037093-50f6-49b9-93c2-f37a7f1290d6\") " pod="openshift-image-registry/node-ca-jhqzq"
Oct 30 16:46:25.722301 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720510    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-etc-openvswitch\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.722301 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720530    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-log-socket\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.722301 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720550    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-sysctl-d\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.722301 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720571    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/92b17f34-17e7-4d4f-b43a-4e828ede0187-hosts-file\") pod \"node-resolver-nlr68\" (UID: \"92b17f34-17e7-4d4f-b43a-4e828ede0187\") " pod="openshift-dns/node-resolver-nlr68"
Oct 30 16:46:25.722301 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720593    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-slash\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.722301 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720616    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhxl7\" (UniqueName: \"kubernetes.io/projected/7bfc195d-6fbc-45c8-9e2a-f9c206838651-kube-api-access-jhxl7\") pod \"network-check-target-r2pxh\" (UID: \"7bfc195d-6fbc-45c8-9e2a-f9c206838651\") " pod="openshift-network-diagnostics/network-check-target-r2pxh"
Oct 30 16:46:25.722301 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720637    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:46:25.722301 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720640    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/60f548f1-6eae-4929-bfd7-db1db9aab72e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:46:25.722301 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720658    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-etc-selinux\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:46:25.722301 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720696    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-run-netns\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.722301 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720718    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-etc-selinux\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:46:25.723153 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720722    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-cni-bin\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.723153 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720755    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6jfp\" (UniqueName: \"kubernetes.io/projected/60f548f1-6eae-4929-bfd7-db1db9aab72e-kube-api-access-k6jfp\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:46:25.723153 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720779    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-run-netns\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.723153 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720783    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-run-netns\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.723153 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720801    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-device-dir\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:46:25.723153 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720831    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d90edf93-884d-46e3-b1f7-dcae29944ca2-ovnkube-config\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.723153 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720857    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-run-k8s-cni-cncf-io\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.723153 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720884    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-multus-socket-dir-parent\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.723153 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720913    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-run-multus-certs\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.723153 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720939    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x97hn\" (UniqueName: \"kubernetes.io/projected/33a051e9-adff-4247-8988-f6ea635aaee0-kube-api-access-x97hn\") pod \"iptables-alerter-26t96\" (UID: \"33a051e9-adff-4247-8988-f6ea635aaee0\") " pod="openshift-network-operator/iptables-alerter-26t96"
Oct 30 16:46:25.723153 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720971    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-lib-modules\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.723153 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720995    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-node-log\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.723153 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.721021    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d90edf93-884d-46e3-b1f7-dcae29944ca2-env-overrides\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.723153 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.721045    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6c03c304-7756-45bb-9597-350026ed361e-agent-certs\") pod \"konnectivity-agent-vw5v7\" (UID: \"6c03c304-7756-45bb-9597-350026ed361e\") " pod="kube-system/konnectivity-agent-vw5v7"
Oct 30 16:46:25.723153 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720217    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60f548f1-6eae-4929-bfd7-db1db9aab72e-cnibin\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:46:25.723153 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.721067    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db037093-50f6-49b9-93c2-f37a7f1290d6-host\") pod \"node-ca-jhqzq\" (UID: \"db037093-50f6-49b9-93c2-f37a7f1290d6\") " pod="openshift-image-registry/node-ca-jhqzq"
Oct 30 16:46:25.723153 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.721091    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-systemd-units\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.723999 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.721137    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-sysconfig\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.723999 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.721151    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-systemd-units\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.723999 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.721181    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/db037093-50f6-49b9-93c2-f37a7f1290d6-serviceca\") pod \"node-ca-jhqzq\" (UID: \"db037093-50f6-49b9-93c2-f37a7f1290d6\") " pod="openshift-image-registry/node-ca-jhqzq"
Oct 30 16:46:25.723999 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.721205    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-etc-openvswitch\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.723999 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.721243    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-var-lib-openvswitch\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.723999 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.721296    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/92b17f34-17e7-4d4f-b43a-4e828ede0187-hosts-file\") pod \"node-resolver-nlr68\" (UID: \"92b17f34-17e7-4d4f-b43a-4e828ede0187\") " pod="openshift-dns/node-resolver-nlr68"
Oct 30 16:46:25.723999 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.721302    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-var-lib-cni-multus\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.723999 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.721394    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-sysctl-d\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.723999 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.721435    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-slash\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.723999 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.721245    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-log-socket\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.723999 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.721543    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/60154025-ad1c-41b0-b28c-1a83e5994685-multus-daemon-config\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.723999 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.721550    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-multus-socket-dir-parent\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.723999 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.721583    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/498ef897-364c-4e19-9fbf-cdc46aac623f-kubelet-config\") pod \"global-pull-secret-syncer-5cgn8\" (UID: \"498ef897-364c-4e19-9fbf-cdc46aac623f\") " pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:46:25.723999 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.721576    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-run-multus-certs\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.723999 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.721609    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:46:25.723999 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.721614    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-system-cni-dir\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.723999 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.721642    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-multus-cni-dir\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.723999 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.721669    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-registration-dir\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:46:25.724868 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.721694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-var-lib-kubelet\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.724868 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.721720    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-run-ovn-kubernetes\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.724868 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.721723    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-lib-modules\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.724868 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.721760    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-sys-fs\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:46:25.724868 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.721780    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/498ef897-364c-4e19-9fbf-cdc46aac623f-kubelet-config\") pod \"global-pull-secret-syncer-5cgn8\" (UID: \"498ef897-364c-4e19-9fbf-cdc46aac623f\") " pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:46:25.724868 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.721794    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdd6c\" (UniqueName: \"kubernetes.io/projected/cf8a28a6-fd11-45dc-9927-adeba7780b46-kube-api-access-vdd6c\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.724868 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.720756    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-cni-bin\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.724868 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.721829    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-system-cni-dir\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.724868 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.721964    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-var-lib-cni-bin\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.724868 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.721990    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-multus-cni-dir\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.724868 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.722049    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-registration-dir\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:46:25.724868 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.722082    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-run-netns\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.724868 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.722100    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-var-lib-kubelet\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.724868 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.722137    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-node-log\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.724868 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.722139    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-device-dir\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:46:25.724868 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.721824    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5wnk\" (UniqueName: \"kubernetes.io/projected/92b17f34-17e7-4d4f-b43a-4e828ede0187-kube-api-access-r5wnk\") pod \"node-resolver-nlr68\" (UID: \"92b17f34-17e7-4d4f-b43a-4e828ede0187\") " pod="openshift-dns/node-resolver-nlr68"
Oct 30 16:46:25.724868 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.722184    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6c03c304-7756-45bb-9597-350026ed361e-konnectivity-ca\") pod \"konnectivity-agent-vw5v7\" (UID: \"6c03c304-7756-45bb-9597-350026ed361e\") " pod="kube-system/konnectivity-agent-vw5v7"
Oct 30 16:46:25.724868 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.722185    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-run-k8s-cni-cncf-io\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.725808 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.722270    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d90edf93-884d-46e3-b1f7-dcae29944ca2-env-overrides\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.725808 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.722325    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-run-ovn-kubernetes\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.725808 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.722375    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-sys-fs\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:46:25.725808 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.722616    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db037093-50f6-49b9-93c2-f37a7f1290d6-host\") pod \"node-ca-jhqzq\" (UID: \"db037093-50f6-49b9-93c2-f37a7f1290d6\") " pod="openshift-image-registry/node-ca-jhqzq"
Oct 30 16:46:25.725808 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.722735    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d90edf93-884d-46e3-b1f7-dcae29944ca2-ovnkube-config\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.725808 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.722743    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6c03c304-7756-45bb-9597-350026ed361e-konnectivity-ca\") pod \"konnectivity-agent-vw5v7\" (UID: \"6c03c304-7756-45bb-9597-350026ed361e\") " pod="kube-system/konnectivity-agent-vw5v7"
Oct 30 16:46:25.725808 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.722812    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-var-lib-kubelet\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.725808 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.723832    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-tuned\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.725808 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.724293    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d90edf93-884d-46e3-b1f7-dcae29944ca2-ovn-node-metrics-cert\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.725808 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.724713    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf8a28a6-fd11-45dc-9927-adeba7780b46-tmp\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.726355 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.726053    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6c03c304-7756-45bb-9597-350026ed361e-agent-certs\") pod \"konnectivity-agent-vw5v7\" (UID: \"6c03c304-7756-45bb-9597-350026ed361e\") " pod="kube-system/konnectivity-agent-vw5v7"
Oct 30 16:46:25.756208 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.756177    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkqgx\" (UniqueName: \"kubernetes.io/projected/db037093-50f6-49b9-93c2-f37a7f1290d6-kube-api-access-mkqgx\") pod \"node-ca-jhqzq\" (UID: \"db037093-50f6-49b9-93c2-f37a7f1290d6\") " pod="openshift-image-registry/node-ca-jhqzq"
Oct 30 16:46:25.760157 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:25.760129    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 16:46:25.760157 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:25.760160    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 16:46:25.760368 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:25.760176    2581 projected.go:196] Error preparing data for projected volume kube-api-access-jhxl7 for pod openshift-network-diagnostics/network-check-target-r2pxh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 16:46:25.760368 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:25.760260    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7bfc195d-6fbc-45c8-9e2a-f9c206838651-kube-api-access-jhxl7 podName:7bfc195d-6fbc-45c8-9e2a-f9c206838651 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:26.260237957 +0000 UTC m=+3.042553223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jhxl7" (UniqueName: "kubernetes.io/projected/7bfc195d-6fbc-45c8-9e2a-f9c206838651-kube-api-access-jhxl7") pod "network-check-target-r2pxh" (UID: "7bfc195d-6fbc-45c8-9e2a-f9c206838651") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 16:46:25.761046 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.760517    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdd6c\" (UniqueName: \"kubernetes.io/projected/cf8a28a6-fd11-45dc-9927-adeba7780b46-kube-api-access-vdd6c\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.761046 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.760996    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5wnk\" (UniqueName: \"kubernetes.io/projected/92b17f34-17e7-4d4f-b43a-4e828ede0187-kube-api-access-r5wnk\") pod \"node-resolver-nlr68\" (UID: \"92b17f34-17e7-4d4f-b43a-4e828ede0187\") " pod="openshift-dns/node-resolver-nlr68"
Oct 30 16:46:25.763322 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.763268    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5l7w\" (UniqueName: \"kubernetes.io/projected/60154025-ad1c-41b0-b28c-1a83e5994685-kube-api-access-f5l7w\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.764055 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.764030    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6jfp\" (UniqueName: \"kubernetes.io/projected/60f548f1-6eae-4929-bfd7-db1db9aab72e-kube-api-access-k6jfp\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:46:25.764397 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.764378    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxv5t\" (UniqueName: \"kubernetes.io/projected/d90edf93-884d-46e3-b1f7-dcae29944ca2-kube-api-access-cxv5t\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.766595 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.766547    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jch5v\" (UniqueName: \"kubernetes.io/projected/980827e6-3d1b-4924-acde-8d408d5e50df-kube-api-access-jch5v\") pod \"network-metrics-daemon-7zf5f\" (UID: \"980827e6-3d1b-4924-acde-8d408d5e50df\") " pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:46:25.769810 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.769784    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgh6f\" (UniqueName: \"kubernetes.io/projected/5cf6407b-d0de-4fa7-a346-1f211ab4c792-kube-api-access-wgh6f\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:46:25.770025 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.770001    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x97hn\" (UniqueName: \"kubernetes.io/projected/33a051e9-adff-4247-8988-f6ea635aaee0-kube-api-access-x97hn\") pod \"iptables-alerter-26t96\" (UID: \"33a051e9-adff-4247-8988-f6ea635aaee0\") " pod="openshift-network-operator/iptables-alerter-26t96"
Oct 30 16:46:25.840160 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.840064    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:46:25.847564 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.847515    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jhqzq"
Oct 30 16:46:25.851398 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:25.851352    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf8a28a6_fd11_45dc_9927_adeba7780b46.slice/crio-c0facc9d32205355d7bdb1da5f5eee80eb0d67b0f77cecc519bfb1f3b16cf2fb WatchSource:0}: Error finding container c0facc9d32205355d7bdb1da5f5eee80eb0d67b0f77cecc519bfb1f3b16cf2fb: Status 404 returned error can't find the container with id c0facc9d32205355d7bdb1da5f5eee80eb0d67b0f77cecc519bfb1f3b16cf2fb
Oct 30 16:46:25.856320 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.856293    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nlr68"
Oct 30 16:46:25.858419 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:25.858370    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb037093_50f6_49b9_93c2_f37a7f1290d6.slice/crio-b7b4c2102de3367edb746a1e9cd1853a704af480c3fd6cb379ae7ab279125927 WatchSource:0}: Error finding container b7b4c2102de3367edb746a1e9cd1853a704af480c3fd6cb379ae7ab279125927: Status 404 returned error can't find the container with id b7b4c2102de3367edb746a1e9cd1853a704af480c3fd6cb379ae7ab279125927
Oct 30 16:46:25.861427 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.861402    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8t9lr"
Oct 30 16:46:25.875519 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:25.875488    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60154025_ad1c_41b0_b28c_1a83e5994685.slice/crio-6fd6497c3149bc16e98b56ab74bcdf05d0cbe81c1705beee194a593125a0dbaa WatchSource:0}: Error finding container 6fd6497c3149bc16e98b56ab74bcdf05d0cbe81c1705beee194a593125a0dbaa: Status 404 returned error can't find the container with id 6fd6497c3149bc16e98b56ab74bcdf05d0cbe81c1705beee194a593125a0dbaa
Oct 30 16:46:25.882098 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.882067    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vw5v7"
Oct 30 16:46:25.893290 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.893258    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:46:25.893982 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:25.893954    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c03c304_7756_45bb_9597_350026ed361e.slice/crio-ee74704cb44e6789528ec6ee598a4d54428d640f5cf90cf6496d0e8b95bbce93 WatchSource:0}: Error finding container ee74704cb44e6789528ec6ee598a4d54428d640f5cf90cf6496d0e8b95bbce93: Status 404 returned error can't find the container with id ee74704cb44e6789528ec6ee598a4d54428d640f5cf90cf6496d0e8b95bbce93
Oct 30 16:46:25.895295 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.895263    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-26t96"
Oct 30 16:46:25.897978 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.897927    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:25.906288 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:25.906119    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf6407b_d0de_4fa7_a346_1f211ab4c792.slice/crio-89af626f67364a3694b3a603d4c9a6ef680340d7a0ae1c6776fa579c07bb61e1 WatchSource:0}: Error finding container 89af626f67364a3694b3a603d4c9a6ef680340d7a0ae1c6776fa579c07bb61e1: Status 404 returned error can't find the container with id 89af626f67364a3694b3a603d4c9a6ef680340d7a0ae1c6776fa579c07bb61e1
Oct 30 16:46:25.910622 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:25.910593    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a051e9_adff_4247_8988_f6ea635aaee0.slice/crio-37db11e32e134f2c89a5397054d650db55928e15f04054186705a8d79f6458a0 WatchSource:0}: Error finding container 37db11e32e134f2c89a5397054d650db55928e15f04054186705a8d79f6458a0: Status 404 returned error can't find the container with id 37db11e32e134f2c89a5397054d650db55928e15f04054186705a8d79f6458a0
Oct 30 16:46:25.913824 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:25.913797    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:46:25.914341 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:25.914213    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd90edf93_884d_46e3_b1f7_dcae29944ca2.slice/crio-b2e04792d09f6d7a2565deeb7ae4fdb20532e1b185bdfe090bb3185640d7d8b8 WatchSource:0}: Error finding container b2e04792d09f6d7a2565deeb7ae4fdb20532e1b185bdfe090bb3185640d7d8b8: Status 404 returned error can't find the container with id b2e04792d09f6d7a2565deeb7ae4fdb20532e1b185bdfe090bb3185640d7d8b8
Oct 30 16:46:25.924930 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:25.924894    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f548f1_6eae_4929_bfd7_db1db9aab72e.slice/crio-01c9e0f62f817b50473b22a6c10db6d80f9c56e0ee401c5e1cd9d5422e5ae272 WatchSource:0}: Error finding container 01c9e0f62f817b50473b22a6c10db6d80f9c56e0ee401c5e1cd9d5422e5ae272: Status 404 returned error can't find the container with id 01c9e0f62f817b50473b22a6c10db6d80f9c56e0ee401c5e1cd9d5422e5ae272
Oct 30 16:46:26.227563 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:26.227525    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs\") pod \"network-metrics-daemon-7zf5f\" (UID: \"980827e6-3d1b-4924-acde-8d408d5e50df\") " pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:46:26.227753 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:26.227586    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/498ef897-364c-4e19-9fbf-cdc46aac623f-original-pull-secret\") pod \"global-pull-secret-syncer-5cgn8\" (UID: \"498ef897-364c-4e19-9fbf-cdc46aac623f\") " pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:46:26.227753 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:26.227713    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 16:46:26.227753 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:26.227718    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 16:46:26.227897 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:26.227777    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/498ef897-364c-4e19-9fbf-cdc46aac623f-original-pull-secret podName:498ef897-364c-4e19-9fbf-cdc46aac623f nodeName:}" failed. No retries permitted until 2026-10-30 16:46:27.227762953 +0000 UTC m=+4.010078219 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/498ef897-364c-4e19-9fbf-cdc46aac623f-original-pull-secret") pod "global-pull-secret-syncer-5cgn8" (UID: "498ef897-364c-4e19-9fbf-cdc46aac623f") : object "kube-system"/"original-pull-secret" not registered
Oct 30 16:46:26.227897 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:26.227794    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs podName:980827e6-3d1b-4924-acde-8d408d5e50df nodeName:}" failed. No retries permitted until 2026-10-30 16:46:27.22778781 +0000 UTC m=+4.010103053 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs") pod "network-metrics-daemon-7zf5f" (UID: "980827e6-3d1b-4924-acde-8d408d5e50df") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 16:46:26.329435 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:26.328850    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhxl7\" (UniqueName: \"kubernetes.io/projected/7bfc195d-6fbc-45c8-9e2a-f9c206838651-kube-api-access-jhxl7\") pod \"network-check-target-r2pxh\" (UID: \"7bfc195d-6fbc-45c8-9e2a-f9c206838651\") " pod="openshift-network-diagnostics/network-check-target-r2pxh"
Oct 30 16:46:26.329435 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:26.329024    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 16:46:26.329435 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:26.329042    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 16:46:26.329435 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:26.329057    2581 projected.go:196] Error preparing data for projected volume kube-api-access-jhxl7 for pod openshift-network-diagnostics/network-check-target-r2pxh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 16:46:26.329435 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:26.329115    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7bfc195d-6fbc-45c8-9e2a-f9c206838651-kube-api-access-jhxl7 podName:7bfc195d-6fbc-45c8-9e2a-f9c206838651 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:27.329100542 +0000 UTC m=+4.111415786 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jhxl7" (UniqueName: "kubernetes.io/projected/7bfc195d-6fbc-45c8-9e2a-f9c206838651-kube-api-access-jhxl7") pod "network-check-target-r2pxh" (UID: "7bfc195d-6fbc-45c8-9e2a-f9c206838651") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 16:46:26.599795 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:26.599124    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 16:41:26 +0000 UTC" deadline="2028-08-09 01:26:07.871671628 +0000 UTC"
Oct 30 16:46:26.599795 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:26.599408    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15560h39m41.272268942s"
Oct 30 16:46:26.653728 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:26.653154    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2pxh"
Oct 30 16:46:26.653728 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:26.653291    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2pxh" podUID="7bfc195d-6fbc-45c8-9e2a-f9c206838651"
Oct 30 16:46:26.678050 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:26.677932    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8t9lr" event={"ID":"60154025-ad1c-41b0-b28c-1a83e5994685","Type":"ContainerStarted","Data":"6fd6497c3149bc16e98b56ab74bcdf05d0cbe81c1705beee194a593125a0dbaa"}
Oct 30 16:46:26.688840 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:26.688091    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-13.ec2.internal" event={"ID":"777e8481b0741b0fafcaf01b060f8274","Type":"ContainerStarted","Data":"aeea35f592c9272b07b8187a29883ea0b66cecac8282b0ab5466ff275d693aa0"}
Oct 30 16:46:26.690616 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:26.690584    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgwcr" event={"ID":"60f548f1-6eae-4929-bfd7-db1db9aab72e","Type":"ContainerStarted","Data":"01c9e0f62f817b50473b22a6c10db6d80f9c56e0ee401c5e1cd9d5422e5ae272"}
Oct 30 16:46:26.692844 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:26.692817    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g" event={"ID":"d90edf93-884d-46e3-b1f7-dcae29944ca2","Type":"ContainerStarted","Data":"b2e04792d09f6d7a2565deeb7ae4fdb20532e1b185bdfe090bb3185640d7d8b8"}
Oct 30 16:46:26.698640 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:26.698607    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vw5v7" event={"ID":"6c03c304-7756-45bb-9597-350026ed361e","Type":"ContainerStarted","Data":"ee74704cb44e6789528ec6ee598a4d54428d640f5cf90cf6496d0e8b95bbce93"}
Oct 30 16:46:26.703229 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:26.703159    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nlr68" event={"ID":"92b17f34-17e7-4d4f-b43a-4e828ede0187","Type":"ContainerStarted","Data":"80ab8702a92038db0be89a001a943928d4826d7222d85b3baad1fce938d6cf46"}
Oct 30 16:46:26.705409 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:26.705361    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jhqzq" event={"ID":"db037093-50f6-49b9-93c2-f37a7f1290d6","Type":"ContainerStarted","Data":"b7b4c2102de3367edb746a1e9cd1853a704af480c3fd6cb379ae7ab279125927"}
Oct 30 16:46:26.711864 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:26.711755    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f8wv9" event={"ID":"cf8a28a6-fd11-45dc-9927-adeba7780b46","Type":"ContainerStarted","Data":"c0facc9d32205355d7bdb1da5f5eee80eb0d67b0f77cecc519bfb1f3b16cf2fb"}
Oct 30 16:46:26.718366 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:26.718307    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-26t96" event={"ID":"33a051e9-adff-4247-8988-f6ea635aaee0","Type":"ContainerStarted","Data":"37db11e32e134f2c89a5397054d650db55928e15f04054186705a8d79f6458a0"}
Oct 30 16:46:26.725114 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:26.725011    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf" event={"ID":"5cf6407b-d0de-4fa7-a346-1f211ab4c792","Type":"ContainerStarted","Data":"89af626f67364a3694b3a603d4c9a6ef680340d7a0ae1c6776fa579c07bb61e1"}
Oct 30 16:46:26.767172 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:26.765336    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-13.ec2.internal" podStartSLOduration=1.7653163379999999 podStartE2EDuration="1.765316338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:46:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:26.764841033 +0000 UTC m=+3.547156300" watchObservedRunningTime="2026-10-30 16:46:26.765316338 +0000 UTC m=+3.547631625"
Oct 30 16:46:27.238008 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:27.237977    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs\") pod \"network-metrics-daemon-7zf5f\" (UID: \"980827e6-3d1b-4924-acde-8d408d5e50df\") " pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:46:27.238166 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:27.238033    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/498ef897-364c-4e19-9fbf-cdc46aac623f-original-pull-secret\") pod \"global-pull-secret-syncer-5cgn8\" (UID: \"498ef897-364c-4e19-9fbf-cdc46aac623f\") " pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:46:27.238229 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:27.238165    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 16:46:27.238284 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:27.238229    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/498ef897-364c-4e19-9fbf-cdc46aac623f-original-pull-secret podName:498ef897-364c-4e19-9fbf-cdc46aac623f nodeName:}" failed. No retries permitted until 2026-10-30 16:46:29.238209504 +0000 UTC m=+6.020524765 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/498ef897-364c-4e19-9fbf-cdc46aac623f-original-pull-secret") pod "global-pull-secret-syncer-5cgn8" (UID: "498ef897-364c-4e19-9fbf-cdc46aac623f") : object "kube-system"/"original-pull-secret" not registered
Oct 30 16:46:27.238351 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:27.238302    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 16:46:27.238351 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:27.238336    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs podName:980827e6-3d1b-4924-acde-8d408d5e50df nodeName:}" failed. No retries permitted until 2026-10-30 16:46:29.238324058 +0000 UTC m=+6.020639318 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs") pod "network-metrics-daemon-7zf5f" (UID: "980827e6-3d1b-4924-acde-8d408d5e50df") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 16:46:27.338896 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:27.338672    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhxl7\" (UniqueName: \"kubernetes.io/projected/7bfc195d-6fbc-45c8-9e2a-f9c206838651-kube-api-access-jhxl7\") pod \"network-check-target-r2pxh\" (UID: \"7bfc195d-6fbc-45c8-9e2a-f9c206838651\") " pod="openshift-network-diagnostics/network-check-target-r2pxh"
Oct 30 16:46:27.339076 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:27.338905    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 16:46:27.339076 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:27.338926    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 16:46:27.339076 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:27.338940    2581 projected.go:196] Error preparing data for projected volume kube-api-access-jhxl7 for pod openshift-network-diagnostics/network-check-target-r2pxh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 16:46:27.339076 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:27.339005    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7bfc195d-6fbc-45c8-9e2a-f9c206838651-kube-api-access-jhxl7 podName:7bfc195d-6fbc-45c8-9e2a-f9c206838651 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:29.338987066 +0000 UTC m=+6.121302334 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jhxl7" (UniqueName: "kubernetes.io/projected/7bfc195d-6fbc-45c8-9e2a-f9c206838651-kube-api-access-jhxl7") pod "network-check-target-r2pxh" (UID: "7bfc195d-6fbc-45c8-9e2a-f9c206838651") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 16:46:27.600963 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:27.600482    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 16:41:26 +0000 UTC" deadline="2028-04-30 18:31:32.658664237 +0000 UTC"
Oct 30 16:46:27.600963 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:27.600544    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13153h45m5.058124605s"
Oct 30 16:46:27.679831 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:27.679802    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:46:27.680000 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:27.679842    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:46:27.680000 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:27.679949    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zf5f" podUID="980827e6-3d1b-4924-acde-8d408d5e50df"
Oct 30 16:46:27.680138 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:27.680024    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cgn8" podUID="498ef897-364c-4e19-9fbf-cdc46aac623f"
Oct 30 16:46:27.680138 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:27.680093    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2pxh"
Oct 30 16:46:27.680248 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:27.680171    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2pxh" podUID="7bfc195d-6fbc-45c8-9e2a-f9c206838651"
Oct 30 16:46:27.752726 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:27.752288    2581 generic.go:356] "Generic (PLEG): container finished" podID="91ca0575ad91df65b05568b935060571" containerID="bb1d71c86b64d98256f38c4eb7ef389b3ae6ccd113341dd92bfb6f29b9d0e0e8" exitCode=0
Oct 30 16:46:27.752726 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:27.752435    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal" event={"ID":"91ca0575ad91df65b05568b935060571","Type":"ContainerDied","Data":"bb1d71c86b64d98256f38c4eb7ef389b3ae6ccd113341dd92bfb6f29b9d0e0e8"}
Oct 30 16:46:28.763046 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:28.762285    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal" event={"ID":"91ca0575ad91df65b05568b935060571","Type":"ContainerStarted","Data":"ef3aafe3b0b4756bfb0492d64c82b88fc04133349acd027a0570f348f86612b6"}
Oct 30 16:46:28.815078 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:28.814784    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal" podStartSLOduration=3.814764194 podStartE2EDuration="3.814764194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:46:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:28.813814479 +0000 UTC m=+5.596129747" watchObservedRunningTime="2026-10-30 16:46:28.814764194 +0000 UTC m=+5.597079462"
Oct 30 16:46:29.257611 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:29.257565    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs\") pod \"network-metrics-daemon-7zf5f\" (UID: \"980827e6-3d1b-4924-acde-8d408d5e50df\") " pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:46:29.257799 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:29.257638    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/498ef897-364c-4e19-9fbf-cdc46aac623f-original-pull-secret\") pod \"global-pull-secret-syncer-5cgn8\" (UID: \"498ef897-364c-4e19-9fbf-cdc46aac623f\") " pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:46:29.257868 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:29.257809    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 16:46:29.257925 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:29.257873    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/498ef897-364c-4e19-9fbf-cdc46aac623f-original-pull-secret podName:498ef897-364c-4e19-9fbf-cdc46aac623f nodeName:}" failed. No retries permitted until 2026-10-30 16:46:33.257855677 +0000 UTC m=+10.040170923 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/498ef897-364c-4e19-9fbf-cdc46aac623f-original-pull-secret") pod "global-pull-secret-syncer-5cgn8" (UID: "498ef897-364c-4e19-9fbf-cdc46aac623f") : object "kube-system"/"original-pull-secret" not registered
Oct 30 16:46:29.258324 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:29.258304    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 16:46:29.258397 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:29.258361    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs podName:980827e6-3d1b-4924-acde-8d408d5e50df nodeName:}" failed. No retries permitted until 2026-10-30 16:46:33.258345635 +0000 UTC m=+10.040660882 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs") pod "network-metrics-daemon-7zf5f" (UID: "980827e6-3d1b-4924-acde-8d408d5e50df") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 16:46:29.358639 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:29.358551    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhxl7\" (UniqueName: \"kubernetes.io/projected/7bfc195d-6fbc-45c8-9e2a-f9c206838651-kube-api-access-jhxl7\") pod \"network-check-target-r2pxh\" (UID: \"7bfc195d-6fbc-45c8-9e2a-f9c206838651\") " pod="openshift-network-diagnostics/network-check-target-r2pxh"
Oct 30 16:46:29.358803 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:29.358784    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 16:46:29.358849 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:29.358806    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 16:46:29.358849 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:29.358820    2581 projected.go:196] Error preparing data for projected volume kube-api-access-jhxl7 for pod openshift-network-diagnostics/network-check-target-r2pxh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 16:46:29.358908 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:29.358885    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7bfc195d-6fbc-45c8-9e2a-f9c206838651-kube-api-access-jhxl7 podName:7bfc195d-6fbc-45c8-9e2a-f9c206838651 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:33.358864114 +0000 UTC m=+10.141179359 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jhxl7" (UniqueName: "kubernetes.io/projected/7bfc195d-6fbc-45c8-9e2a-f9c206838651-kube-api-access-jhxl7") pod "network-check-target-r2pxh" (UID: "7bfc195d-6fbc-45c8-9e2a-f9c206838651") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 16:46:29.657582 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:29.657468    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:46:29.657768 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:29.657639    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zf5f" podUID="980827e6-3d1b-4924-acde-8d408d5e50df"
Oct 30 16:46:29.658000 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:29.657979    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:46:29.658108 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:29.658054    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cgn8" podUID="498ef897-364c-4e19-9fbf-cdc46aac623f"
Oct 30 16:46:29.658108 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:29.658090    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2pxh"
Oct 30 16:46:29.658216 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:29.658131    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2pxh" podUID="7bfc195d-6fbc-45c8-9e2a-f9c206838651"
Oct 30 16:46:29.769157 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:29.769115    2581 generic.go:356] "Generic (PLEG): container finished" podID="60f548f1-6eae-4929-bfd7-db1db9aab72e" containerID="6cd0f0e8e2e1dcd5c2fcb9dbaca2dfa3b2aaf55c58a588dd17f7f191f2615274" exitCode=0
Oct 30 16:46:29.770589 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:29.770340    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgwcr" event={"ID":"60f548f1-6eae-4929-bfd7-db1db9aab72e","Type":"ContainerDied","Data":"6cd0f0e8e2e1dcd5c2fcb9dbaca2dfa3b2aaf55c58a588dd17f7f191f2615274"}
Oct 30 16:46:30.775184 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:30.774545    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vw5v7" event={"ID":"6c03c304-7756-45bb-9597-350026ed361e","Type":"ContainerStarted","Data":"4bcc82077d7d80598325a0e75b925c0fdcf1cae3cb494955a662b8a7a1f54f51"}
Oct 30 16:46:31.669568 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:31.668682    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:46:31.669568 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:31.668828    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zf5f" podUID="980827e6-3d1b-4924-acde-8d408d5e50df"
Oct 30 16:46:31.669568 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:31.669244    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:46:31.669568 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:31.669339    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cgn8" podUID="498ef897-364c-4e19-9fbf-cdc46aac623f"
Oct 30 16:46:31.669568 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:31.669397    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2pxh"
Oct 30 16:46:31.669568 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:31.669480    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2pxh" podUID="7bfc195d-6fbc-45c8-9e2a-f9c206838651"
Oct 30 16:46:31.781247 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:31.781205    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jhqzq" event={"ID":"db037093-50f6-49b9-93c2-f37a7f1290d6","Type":"ContainerStarted","Data":"f58fa9935afefc5248b426b5d9240059bf164aacbb6f78ab12d3a0f6374085c8"}
Oct 30 16:46:31.834253 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:31.833674    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vw5v7" podStartSLOduration=3.775081971 podStartE2EDuration="7.833651759s" totalImagesPullingTime="4.058569788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:30.825727096 +0000 UTC m=+7.608042364" watchObservedRunningTime="2026-10-30 16:46:31.833651759 +0000 UTC m=+8.615967028"
Oct 30 16:46:32.786438 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:32.786399    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf" event={"ID":"5cf6407b-d0de-4fa7-a346-1f211ab4c792","Type":"ContainerStarted","Data":"6e0a5a756cf80c4d05fa81f3b71f9eaccb44ec8fad2db98344bcf193246ae3cf"}
Oct 30 16:46:33.292100 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:33.292006    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs\") pod \"network-metrics-daemon-7zf5f\" (UID: \"980827e6-3d1b-4924-acde-8d408d5e50df\") " pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:46:33.292100 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:33.292058    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/498ef897-364c-4e19-9fbf-cdc46aac623f-original-pull-secret\") pod \"global-pull-secret-syncer-5cgn8\" (UID: \"498ef897-364c-4e19-9fbf-cdc46aac623f\") " pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:46:33.292356 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:33.292175    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 16:46:33.292356 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:33.292231    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/498ef897-364c-4e19-9fbf-cdc46aac623f-original-pull-secret podName:498ef897-364c-4e19-9fbf-cdc46aac623f nodeName:}" failed. No retries permitted until 2026-10-30 16:46:41.292215663 +0000 UTC m=+18.074530917 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/498ef897-364c-4e19-9fbf-cdc46aac623f-original-pull-secret") pod "global-pull-secret-syncer-5cgn8" (UID: "498ef897-364c-4e19-9fbf-cdc46aac623f") : object "kube-system"/"original-pull-secret" not registered
Oct 30 16:46:33.292808 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:33.292622    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 16:46:33.292808 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:33.292670    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs podName:980827e6-3d1b-4924-acde-8d408d5e50df nodeName:}" failed. No retries permitted until 2026-10-30 16:46:41.29265721 +0000 UTC m=+18.074972465 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs") pod "network-metrics-daemon-7zf5f" (UID: "980827e6-3d1b-4924-acde-8d408d5e50df") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 16:46:33.393604 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:33.392926    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhxl7\" (UniqueName: \"kubernetes.io/projected/7bfc195d-6fbc-45c8-9e2a-f9c206838651-kube-api-access-jhxl7\") pod \"network-check-target-r2pxh\" (UID: \"7bfc195d-6fbc-45c8-9e2a-f9c206838651\") " pod="openshift-network-diagnostics/network-check-target-r2pxh"
Oct 30 16:46:33.393604 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:33.393147    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 16:46:33.393604 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:33.393165    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 16:46:33.393604 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:33.393177    2581 projected.go:196] Error preparing data for projected volume kube-api-access-jhxl7 for pod openshift-network-diagnostics/network-check-target-r2pxh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 16:46:33.393604 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:33.393233    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7bfc195d-6fbc-45c8-9e2a-f9c206838651-kube-api-access-jhxl7 podName:7bfc195d-6fbc-45c8-9e2a-f9c206838651 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:41.393217311 +0000 UTC m=+18.175532566 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jhxl7" (UniqueName: "kubernetes.io/projected/7bfc195d-6fbc-45c8-9e2a-f9c206838651-kube-api-access-jhxl7") pod "network-check-target-r2pxh" (UID: "7bfc195d-6fbc-45c8-9e2a-f9c206838651") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 16:46:33.462686 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:33.462030    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vw5v7"
Oct 30 16:46:33.463266 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:33.463084    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vw5v7"
Oct 30 16:46:33.514498 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:33.514448    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jhqzq" podStartSLOduration=4.327427868 podStartE2EDuration="9.514428611s" totalImagesPullingTime="5.187000743s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:31.834118513 +0000 UTC m=+8.616433784" watchObservedRunningTime="2026-10-30 16:46:33.514428611 +0000 UTC m=+10.296743871"
Oct 30 16:46:33.658099 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:33.658070    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:46:33.658238 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:33.658196    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zf5f" podUID="980827e6-3d1b-4924-acde-8d408d5e50df"
Oct 30 16:46:33.658659 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:33.658637    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:46:33.658819 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:33.658735    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cgn8" podUID="498ef897-364c-4e19-9fbf-cdc46aac623f"
Oct 30 16:46:33.658910 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:33.658843    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2pxh"
Oct 30 16:46:33.658962 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:33.658928    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2pxh" podUID="7bfc195d-6fbc-45c8-9e2a-f9c206838651"
Oct 30 16:46:33.694046 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:33.694017    2581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 16:46:33.794850 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:33.794801    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf" event={"ID":"5cf6407b-d0de-4fa7-a346-1f211ab4c792","Type":"ContainerStarted","Data":"6dfc9b8bb483a406e82322f1d866a1dc7dd76a084555422c430e5715eeac0aea"}
Oct 30 16:46:33.795417 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:33.794873    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vw5v7"
Oct 30 16:46:33.795479 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:33.795425    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vw5v7"
Oct 30 16:46:34.577911 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:34.576535    2581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T16:46:33.694036746Z","UUID":"653e0ee5-545e-4658-b44f-922a86f8612b","Handler":null,"Name":"","Endpoint":""}
Oct 30 16:46:34.579935 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:34.579275    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 16:46:34.579935 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:34.579308    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 16:46:34.798541 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:34.798509    2581 generic.go:356] "Generic (PLEG): container finished" podID="60f548f1-6eae-4929-bfd7-db1db9aab72e" containerID="20a55b69d35871705839d6940e5be751878d235ff075d35aaa46d59330436ab7" exitCode=0
Oct 30 16:46:34.798924 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:34.798608    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgwcr" event={"ID":"60f548f1-6eae-4929-bfd7-db1db9aab72e","Type":"ContainerDied","Data":"20a55b69d35871705839d6940e5be751878d235ff075d35aaa46d59330436ab7"}
Oct 30 16:46:34.801146 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:34.801118    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nlr68" event={"ID":"92b17f34-17e7-4d4f-b43a-4e828ede0187","Type":"ContainerStarted","Data":"56f69d2e6773e9671e2ce506ca93b8bfaa7ebf73489cd736a42d262a49b7f91a"}
Oct 30 16:46:34.803721 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:34.803694    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8t9lr" event={"ID":"60154025-ad1c-41b0-b28c-1a83e5994685","Type":"ContainerStarted","Data":"32317a783e4cdd9b9cd48d0b717a265718ec5459749fd0545a124cedb4efb84f"}
Oct 30 16:46:34.903021 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:34.902956    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-nlr68" podStartSLOduration=2.512697339 podStartE2EDuration="10.902938361s" totalImagesPullingTime="8.390241022s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:34.902654797 +0000 UTC m=+11.684970078" watchObservedRunningTime="2026-10-30 16:46:34.902938361 +0000 UTC m=+11.685253624"
Oct 30 16:46:35.653245 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:35.653156    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:46:35.653245 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:35.653184    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2pxh"
Oct 30 16:46:35.653245 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:35.653171    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:46:35.653508 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:35.653310    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zf5f" podUID="980827e6-3d1b-4924-acde-8d408d5e50df"
Oct 30 16:46:35.653508 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:35.653453    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2pxh" podUID="7bfc195d-6fbc-45c8-9e2a-f9c206838651"
Oct 30 16:46:35.653610 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:35.653545    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cgn8" podUID="498ef897-364c-4e19-9fbf-cdc46aac623f"
Oct 30 16:46:35.817573 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:35.817532    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf" event={"ID":"5cf6407b-d0de-4fa7-a346-1f211ab4c792","Type":"ContainerStarted","Data":"0c7cce84ef0f4eaee46ed574eaebf46a8a55e5a67164f296cbea73e8438afc36"}
Oct 30 16:46:35.867918 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:35.867862    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-8t9lr" podStartSLOduration=3.687422041 podStartE2EDuration="11.86784102s" totalImagesPullingTime="8.180418979s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:34.953591569 +0000 UTC m=+11.735906832" watchObservedRunningTime="2026-10-30 16:46:35.86784102 +0000 UTC m=+12.650156278"
Oct 30 16:46:36.821981 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:36.821903    2581 generic.go:356] "Generic (PLEG): container finished" podID="60f548f1-6eae-4929-bfd7-db1db9aab72e" containerID="e66caa8bf7c7b75c3c66a8c42b5c993bc89d740402b8707e0e638301df6b99db" exitCode=0
Oct 30 16:46:36.822704 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:36.821983    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgwcr" event={"ID":"60f548f1-6eae-4929-bfd7-db1db9aab72e","Type":"ContainerDied","Data":"e66caa8bf7c7b75c3c66a8c42b5c993bc89d740402b8707e0e638301df6b99db"}
Oct 30 16:46:36.823912 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:36.823884    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f8wv9" event={"ID":"cf8a28a6-fd11-45dc-9927-adeba7780b46","Type":"ContainerStarted","Data":"8e483dac05db7856ea8f627cf6fba98785dd8f810425d888aa1c1fc27704935c"}
Oct 30 16:46:36.825448 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:36.825428    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-26t96" event={"ID":"33a051e9-adff-4247-8988-f6ea635aaee0","Type":"ContainerStarted","Data":"4ef2c8b78effd3c9f1fbd41742b870009187102a808daf23b5287ba4c6258141"}
Oct 30 16:46:36.875025 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:36.874957    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf" podStartSLOduration=4.426464566 podStartE2EDuration="12.874942424s" totalImagesPullingTime="8.448477858s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:46:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:35.868696146 +0000 UTC m=+12.651011410" watchObservedRunningTime="2026-10-30 16:46:36.874942424 +0000 UTC m=+13.657257733"
Oct 30 16:46:36.922336 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:36.922271    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-26t96" podStartSLOduration=4.41097625 podStartE2EDuration="12.922253073s" totalImagesPullingTime="8.511276823s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:36.920394568 +0000 UTC m=+13.702709831" watchObservedRunningTime="2026-10-30 16:46:36.922253073 +0000 UTC m=+13.704568337"
Oct 30 16:46:36.968571 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:36.968494    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-f8wv9" podStartSLOduration=2.253697213 podStartE2EDuration="12.968477076s" totalImagesPullingTime="10.714779863s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:36.968181287 +0000 UTC m=+13.750496551" watchObservedRunningTime="2026-10-30 16:46:36.968477076 +0000 UTC m=+13.750792337"
Oct 30 16:46:37.652817 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:37.652630    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2pxh"
Oct 30 16:46:37.652817 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:37.652652    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:46:37.652817 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:37.652757    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2pxh" podUID="7bfc195d-6fbc-45c8-9e2a-f9c206838651"
Oct 30 16:46:37.652817 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:37.652793    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:46:37.653064 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:37.652895    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cgn8" podUID="498ef897-364c-4e19-9fbf-cdc46aac623f"
Oct 30 16:46:37.653064 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:37.653013    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zf5f" podUID="980827e6-3d1b-4924-acde-8d408d5e50df"
Oct 30 16:46:37.830242 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:37.830160    2581 generic.go:356] "Generic (PLEG): container finished" podID="60f548f1-6eae-4929-bfd7-db1db9aab72e" containerID="ec61f1997e8281023d67634750348f941347f0c9c0cdc19836184422061e4339" exitCode=0
Oct 30 16:46:37.830874 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:37.830456    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgwcr" event={"ID":"60f548f1-6eae-4929-bfd7-db1db9aab72e","Type":"ContainerDied","Data":"ec61f1997e8281023d67634750348f941347f0c9c0cdc19836184422061e4339"}
Oct 30 16:46:39.657051 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:39.656838    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2pxh"
Oct 30 16:46:39.657490 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:39.656902    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:46:39.657490 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:39.657141    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2pxh" podUID="7bfc195d-6fbc-45c8-9e2a-f9c206838651"
Oct 30 16:46:39.657490 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:39.656914    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:46:39.657490 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:39.657221    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cgn8" podUID="498ef897-364c-4e19-9fbf-cdc46aac623f"
Oct 30 16:46:39.657490 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:39.657331    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zf5f" podUID="980827e6-3d1b-4924-acde-8d408d5e50df"
Oct 30 16:46:39.839832 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:39.839788    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgwcr" event={"ID":"60f548f1-6eae-4929-bfd7-db1db9aab72e","Type":"ContainerStarted","Data":"f9977fff0fcb6a43c42667ce17ee9dedb0e3e66194bc60a7d71209006a227dc7"}
Oct 30 16:46:40.844052 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:40.844013    2581 generic.go:356] "Generic (PLEG): container finished" podID="60f548f1-6eae-4929-bfd7-db1db9aab72e" containerID="f9977fff0fcb6a43c42667ce17ee9dedb0e3e66194bc60a7d71209006a227dc7" exitCode=0
Oct 30 16:46:40.844480 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:40.844070    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgwcr" event={"ID":"60f548f1-6eae-4929-bfd7-db1db9aab72e","Type":"ContainerDied","Data":"f9977fff0fcb6a43c42667ce17ee9dedb0e3e66194bc60a7d71209006a227dc7"}
Oct 30 16:46:41.353481 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:41.353273    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs\") pod \"network-metrics-daemon-7zf5f\" (UID: \"980827e6-3d1b-4924-acde-8d408d5e50df\") " pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:46:41.353583 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:41.353508    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/498ef897-364c-4e19-9fbf-cdc46aac623f-original-pull-secret\") pod \"global-pull-secret-syncer-5cgn8\" (UID: \"498ef897-364c-4e19-9fbf-cdc46aac623f\") " pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:46:41.353583 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:41.353425    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 16:46:41.353656 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:41.353585    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 16:46:41.353656 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:41.353625    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs podName:980827e6-3d1b-4924-acde-8d408d5e50df nodeName:}" failed. No retries permitted until 2026-10-30 16:46:57.353609131 +0000 UTC m=+34.135924374 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs") pod "network-metrics-daemon-7zf5f" (UID: "980827e6-3d1b-4924-acde-8d408d5e50df") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 16:46:41.353656 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:41.353639    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/498ef897-364c-4e19-9fbf-cdc46aac623f-original-pull-secret podName:498ef897-364c-4e19-9fbf-cdc46aac623f nodeName:}" failed. No retries permitted until 2026-10-30 16:46:57.353633797 +0000 UTC m=+34.135949037 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/498ef897-364c-4e19-9fbf-cdc46aac623f-original-pull-secret") pod "global-pull-secret-syncer-5cgn8" (UID: "498ef897-364c-4e19-9fbf-cdc46aac623f") : object "kube-system"/"original-pull-secret" not registered
Oct 30 16:46:41.454385 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:41.454351    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhxl7\" (UniqueName: \"kubernetes.io/projected/7bfc195d-6fbc-45c8-9e2a-f9c206838651-kube-api-access-jhxl7\") pod \"network-check-target-r2pxh\" (UID: \"7bfc195d-6fbc-45c8-9e2a-f9c206838651\") " pod="openshift-network-diagnostics/network-check-target-r2pxh"
Oct 30 16:46:41.454538 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:41.454523    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 16:46:41.454574 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:41.454545    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 16:46:41.454574 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:41.454555    2581 projected.go:196] Error preparing data for projected volume kube-api-access-jhxl7 for pod openshift-network-diagnostics/network-check-target-r2pxh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 16:46:41.454634 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:41.454603    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7bfc195d-6fbc-45c8-9e2a-f9c206838651-kube-api-access-jhxl7 podName:7bfc195d-6fbc-45c8-9e2a-f9c206838651 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:57.454591149 +0000 UTC m=+34.236906394 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jhxl7" (UniqueName: "kubernetes.io/projected/7bfc195d-6fbc-45c8-9e2a-f9c206838651-kube-api-access-jhxl7") pod "network-check-target-r2pxh" (UID: "7bfc195d-6fbc-45c8-9e2a-f9c206838651") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 16:46:41.652724 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:41.652641    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:46:41.652843 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:41.652648    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2pxh"
Oct 30 16:46:41.652843 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:41.652770    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:46:41.652908 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:41.652836    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2pxh" podUID="7bfc195d-6fbc-45c8-9e2a-f9c206838651"
Oct 30 16:46:41.652908 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:41.652753    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zf5f" podUID="980827e6-3d1b-4924-acde-8d408d5e50df"
Oct 30 16:46:41.652990 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:41.652913    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cgn8" podUID="498ef897-364c-4e19-9fbf-cdc46aac623f"
Oct 30 16:46:41.851255 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:41.851222    2581 generic.go:356] "Generic (PLEG): container finished" podID="60f548f1-6eae-4929-bfd7-db1db9aab72e" containerID="ac43b781f034b7d014799164cf0a166002f549398090db6b29bb2cbd9fd22508" exitCode=0
Oct 30 16:46:41.851725 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:41.851308    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgwcr" event={"ID":"60f548f1-6eae-4929-bfd7-db1db9aab72e","Type":"ContainerDied","Data":"ac43b781f034b7d014799164cf0a166002f549398090db6b29bb2cbd9fd22508"}
Oct 30 16:46:41.854997 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:41.854978    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g" event={"ID":"d90edf93-884d-46e3-b1f7-dcae29944ca2","Type":"ContainerStarted","Data":"85b23348e3f7065ddd5567f4452d9fc933fbe8edc4525610f526c23d1614643c"}
Oct 30 16:46:41.855073 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:41.855032    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g" event={"ID":"d90edf93-884d-46e3-b1f7-dcae29944ca2","Type":"ContainerStarted","Data":"32cc948d45ab0663e1ac420bb96f2f6c93db628f04c9dcc8c8855eeed47abf8c"}
Oct 30 16:46:41.855073 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:41.855061    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g" event={"ID":"d90edf93-884d-46e3-b1f7-dcae29944ca2","Type":"ContainerStarted","Data":"6792d203ed7aaf8df55b9badb098e69b0c3e4fc213e956a4bb9724345ea1a02d"}
Oct 30 16:46:41.855143 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:41.855088    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g" event={"ID":"d90edf93-884d-46e3-b1f7-dcae29944ca2","Type":"ContainerStarted","Data":"8598ae51ca3921b96c58899dfb8d18992c25a8b38fd1aa9d16c46563496faa8d"}
Oct 30 16:46:41.855143 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:41.855115    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g" event={"ID":"d90edf93-884d-46e3-b1f7-dcae29944ca2","Type":"ContainerStarted","Data":"fbef3a28f0f673fa4ca5124c339c627d79cf80d4a5d470544206b68de4c3ac8a"}
Oct 30 16:46:41.855143 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:41.855141    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g" event={"ID":"d90edf93-884d-46e3-b1f7-dcae29944ca2","Type":"ContainerStarted","Data":"7e2b779a8653e9625521470cc70ae955b7442559e40c238d9961b9c1af3b31f9"}
Oct 30 16:46:42.860784 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:42.860745    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgwcr" event={"ID":"60f548f1-6eae-4929-bfd7-db1db9aab72e","Type":"ContainerStarted","Data":"7da577c22a41ebc2252c47d48f7b56cba5426c3fbce7ac9b80433b4e9363082e"}
Oct 30 16:46:42.943721 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:42.943670    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bgwcr" podStartSLOduration=7.719095076 podStartE2EDuration="18.943655674s" totalImagesPullingTime="11.224560598s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:46:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:42.943556077 +0000 UTC m=+19.725871339" watchObservedRunningTime="2026-10-30 16:46:42.943655674 +0000 UTC m=+19.725970938"
Oct 30 16:46:43.653929 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:43.653891    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:46:43.654106 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:43.653996    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cgn8" podUID="498ef897-364c-4e19-9fbf-cdc46aac623f"
Oct 30 16:46:43.654106 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:43.654008    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2pxh"
Oct 30 16:46:43.654106 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:43.654036    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:46:43.654106 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:43.654082    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2pxh" podUID="7bfc195d-6fbc-45c8-9e2a-f9c206838651"
Oct 30 16:46:43.654301 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:43.654187    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zf5f" podUID="980827e6-3d1b-4924-acde-8d408d5e50df"
Oct 30 16:46:43.866326 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:43.866293    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g" event={"ID":"d90edf93-884d-46e3-b1f7-dcae29944ca2","Type":"ContainerStarted","Data":"44b44396e684d02fb0f05270dbd1965916e9bf56bc3825f77be6d489af3d4fa8"}
Oct 30 16:46:45.652755 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:45.652721    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2pxh"
Oct 30 16:46:45.653369 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:45.652722    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:46:45.653369 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:45.652852    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2pxh" podUID="7bfc195d-6fbc-45c8-9e2a-f9c206838651"
Oct 30 16:46:45.653369 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:45.652866    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:46:45.653369 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:45.652944    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cgn8" podUID="498ef897-364c-4e19-9fbf-cdc46aac623f"
Oct 30 16:46:45.653369 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:45.653050    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zf5f" podUID="980827e6-3d1b-4924-acde-8d408d5e50df"
Oct 30 16:46:45.874178 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:45.874094    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g" event={"ID":"d90edf93-884d-46e3-b1f7-dcae29944ca2","Type":"ContainerStarted","Data":"4c967d5c343c961d2380a67251b93b9815f21f6f81c6a5d574083736edaba368"}
Oct 30 16:46:45.874350 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:45.874329    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:45.874485 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:45.874417    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:45.889704 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:45.889677    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:45.994026 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:45.993977    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g" podStartSLOduration=6.876492834 podStartE2EDuration="21.993965707s" totalImagesPullingTime="15.117472873s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:45.993674192 +0000 UTC m=+22.775989456" watchObservedRunningTime="2026-10-30 16:46:45.993965707 +0000 UTC m=+22.776280969"
Oct 30 16:46:46.877819 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:46.877789    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:46.897518 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:46.897495    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:46:47.589286 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:47.589214    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r2pxh"]
Oct 30 16:46:47.589459 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:47.589344    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2pxh"
Oct 30 16:46:47.589459 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:47.589433    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2pxh" podUID="7bfc195d-6fbc-45c8-9e2a-f9c206838651"
Oct 30 16:46:47.594398 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:47.594371    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5cgn8"]
Oct 30 16:46:47.594535 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:47.594512    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:46:47.594609 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:47.594593    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cgn8" podUID="498ef897-364c-4e19-9fbf-cdc46aac623f"
Oct 30 16:46:47.594841 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:47.594823    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7zf5f"]
Oct 30 16:46:47.594921 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:47.594912    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:46:47.594998 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:47.594983    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zf5f" podUID="980827e6-3d1b-4924-acde-8d408d5e50df"
Oct 30 16:46:49.652753 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:49.652719    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:46:49.653137 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:49.652771    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2pxh"
Oct 30 16:46:49.653137 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:49.652719    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:46:49.653137 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:49.652865    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zf5f" podUID="980827e6-3d1b-4924-acde-8d408d5e50df"
Oct 30 16:46:49.653137 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:49.652909    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cgn8" podUID="498ef897-364c-4e19-9fbf-cdc46aac623f"
Oct 30 16:46:49.653137 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:49.652993    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2pxh" podUID="7bfc195d-6fbc-45c8-9e2a-f9c206838651"
Oct 30 16:46:51.652723 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:51.652684    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2pxh"
Oct 30 16:46:51.652723 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:51.652710    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:46:51.653245 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:51.652832    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2pxh" podUID="7bfc195d-6fbc-45c8-9e2a-f9c206838651"
Oct 30 16:46:51.653245 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:51.652878    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:46:51.653245 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:51.652961    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cgn8" podUID="498ef897-364c-4e19-9fbf-cdc46aac623f"
Oct 30 16:46:51.653245 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:51.653025    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zf5f" podUID="980827e6-3d1b-4924-acde-8d408d5e50df"
Oct 30 16:46:53.653629 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:53.653462    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:46:53.654071 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:53.653685    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5cgn8" podUID="498ef897-364c-4e19-9fbf-cdc46aac623f"
Oct 30 16:46:53.654071 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:53.653548    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2pxh"
Oct 30 16:46:53.654071 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:53.653532    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:46:53.654071 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:53.653744    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r2pxh" podUID="7bfc195d-6fbc-45c8-9e2a-f9c206838651"
Oct 30 16:46:53.654071 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:53.653828    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zf5f" podUID="980827e6-3d1b-4924-acde-8d408d5e50df"
Oct 30 16:46:54.051360 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.051333    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-13.ec2.internal" event="NodeReady"
Oct 30 16:46:54.051546 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.051489    2581 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 16:46:54.130047 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.130020    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ltvcq"]
Oct 30 16:46:54.133075 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.133059    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ltvcq"
Oct 30 16:46:54.138136 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.138113    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5ztj2"]
Oct 30 16:46:54.143165 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.143143    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5ztj2"
Oct 30 16:46:54.152731 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.152714    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ffzh4\""
Oct 30 16:46:54.152731 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.152722    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 16:46:54.158342 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.158323    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 16:46:54.160974 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.160955    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ltvcq"]
Oct 30 16:46:54.165492 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.165471    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5ztj2"]
Oct 30 16:46:54.244030 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.243994    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03a9bc00-f973-4d50-ac32-a4ca843b977b-metrics-tls\") pod \"dns-default-5ztj2\" (UID: \"03a9bc00-f973-4d50-ac32-a4ca843b977b\") " pod="openshift-dns/dns-default-5ztj2"
Oct 30 16:46:54.244030 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.244030    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/03a9bc00-f973-4d50-ac32-a4ca843b977b-config-volume\") pod \"dns-default-5ztj2\" (UID: \"03a9bc00-f973-4d50-ac32-a4ca843b977b\") " pod="openshift-dns/dns-default-5ztj2"
Oct 30 16:46:54.244231 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.244050    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2fbm\" (UniqueName: \"kubernetes.io/projected/47bd3f72-e568-4ded-ae3f-398eceff7033-kube-api-access-j2fbm\") pod \"ingress-canary-ltvcq\" (UID: \"47bd3f72-e568-4ded-ae3f-398eceff7033\") " pod="openshift-ingress-canary/ingress-canary-ltvcq"
Oct 30 16:46:54.244231 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.244142    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47bd3f72-e568-4ded-ae3f-398eceff7033-cert\") pod \"ingress-canary-ltvcq\" (UID: \"47bd3f72-e568-4ded-ae3f-398eceff7033\") " pod="openshift-ingress-canary/ingress-canary-ltvcq"
Oct 30 16:46:54.244231 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.244165    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03a9bc00-f973-4d50-ac32-a4ca843b977b-tmp-dir\") pod \"dns-default-5ztj2\" (UID: \"03a9bc00-f973-4d50-ac32-a4ca843b977b\") " pod="openshift-dns/dns-default-5ztj2"
Oct 30 16:46:54.244231 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.244180    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxngf\" (UniqueName: \"kubernetes.io/projected/03a9bc00-f973-4d50-ac32-a4ca843b977b-kube-api-access-pxngf\") pod \"dns-default-5ztj2\" (UID: \"03a9bc00-f973-4d50-ac32-a4ca843b977b\") " pod="openshift-dns/dns-default-5ztj2"
Oct 30 16:46:54.252780 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.252757    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 16:46:54.252780 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.252774    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 16:46:54.252867 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.252810    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 16:46:54.252964 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.252952    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f9wpv\""
Oct 30 16:46:54.345450 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.345366    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47bd3f72-e568-4ded-ae3f-398eceff7033-cert\") pod \"ingress-canary-ltvcq\" (UID: \"47bd3f72-e568-4ded-ae3f-398eceff7033\") " pod="openshift-ingress-canary/ingress-canary-ltvcq"
Oct 30 16:46:54.345450 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.345417    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03a9bc00-f973-4d50-ac32-a4ca843b977b-tmp-dir\") pod \"dns-default-5ztj2\" (UID: \"03a9bc00-f973-4d50-ac32-a4ca843b977b\") " pod="openshift-dns/dns-default-5ztj2"
Oct 30 16:46:54.345450 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.345434    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxngf\" (UniqueName: \"kubernetes.io/projected/03a9bc00-f973-4d50-ac32-a4ca843b977b-kube-api-access-pxngf\") pod \"dns-default-5ztj2\" (UID: \"03a9bc00-f973-4d50-ac32-a4ca843b977b\") " pod="openshift-dns/dns-default-5ztj2"
Oct 30 16:46:54.345655 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.345490    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03a9bc00-f973-4d50-ac32-a4ca843b977b-metrics-tls\") pod \"dns-default-5ztj2\" (UID: \"03a9bc00-f973-4d50-ac32-a4ca843b977b\") " pod="openshift-dns/dns-default-5ztj2"
Oct 30 16:46:54.345655 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.345509    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/03a9bc00-f973-4d50-ac32-a4ca843b977b-config-volume\") pod \"dns-default-5ztj2\" (UID: \"03a9bc00-f973-4d50-ac32-a4ca843b977b\") " pod="openshift-dns/dns-default-5ztj2"
Oct 30 16:46:54.345655 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.345540    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2fbm\" (UniqueName: \"kubernetes.io/projected/47bd3f72-e568-4ded-ae3f-398eceff7033-kube-api-access-j2fbm\") pod \"ingress-canary-ltvcq\" (UID: \"47bd3f72-e568-4ded-ae3f-398eceff7033\") " pod="openshift-ingress-canary/ingress-canary-ltvcq"
Oct 30 16:46:54.345655 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:54.345627    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 16:46:54.345814 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:54.345697    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03a9bc00-f973-4d50-ac32-a4ca843b977b-metrics-tls podName:03a9bc00-f973-4d50-ac32-a4ca843b977b nodeName:}" failed. No retries permitted until 2026-10-30 16:46:54.845675695 +0000 UTC m=+31.627990951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/03a9bc00-f973-4d50-ac32-a4ca843b977b-metrics-tls") pod "dns-default-5ztj2" (UID: "03a9bc00-f973-4d50-ac32-a4ca843b977b") : secret "dns-default-metrics-tls" not found
Oct 30 16:46:54.346052 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.346030    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03a9bc00-f973-4d50-ac32-a4ca843b977b-tmp-dir\") pod \"dns-default-5ztj2\" (UID: \"03a9bc00-f973-4d50-ac32-a4ca843b977b\") " pod="openshift-dns/dns-default-5ztj2"
Oct 30 16:46:54.346242 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.346224    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/03a9bc00-f973-4d50-ac32-a4ca843b977b-config-volume\") pod \"dns-default-5ztj2\" (UID: \"03a9bc00-f973-4d50-ac32-a4ca843b977b\") " pod="openshift-dns/dns-default-5ztj2"
Oct 30 16:46:54.349460 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.349444    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47bd3f72-e568-4ded-ae3f-398eceff7033-cert\") pod \"ingress-canary-ltvcq\" (UID: \"47bd3f72-e568-4ded-ae3f-398eceff7033\") " pod="openshift-ingress-canary/ingress-canary-ltvcq"
Oct 30 16:46:54.372644 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.372619    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2fbm\" (UniqueName: \"kubernetes.io/projected/47bd3f72-e568-4ded-ae3f-398eceff7033-kube-api-access-j2fbm\") pod \"ingress-canary-ltvcq\" (UID: \"47bd3f72-e568-4ded-ae3f-398eceff7033\") " pod="openshift-ingress-canary/ingress-canary-ltvcq"
Oct 30 16:46:54.378322 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.378298    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxngf\" (UniqueName: \"kubernetes.io/projected/03a9bc00-f973-4d50-ac32-a4ca843b977b-kube-api-access-pxngf\") pod \"dns-default-5ztj2\" (UID: \"03a9bc00-f973-4d50-ac32-a4ca843b977b\") " pod="openshift-dns/dns-default-5ztj2"
Oct 30 16:46:54.447572 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.447538    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ltvcq"
Oct 30 16:46:54.759816 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.759789    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ltvcq"]
Oct 30 16:46:54.762138 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:54.762120    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47bd3f72_e568_4ded_ae3f_398eceff7033.slice/crio-97014c2ed0aa27e2a90ced34214c7553866b01035fa740207826af713c3049ad WatchSource:0}: Error finding container 97014c2ed0aa27e2a90ced34214c7553866b01035fa740207826af713c3049ad: Status 404 returned error can't find the container with id 97014c2ed0aa27e2a90ced34214c7553866b01035fa740207826af713c3049ad
Oct 30 16:46:54.849043 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.849015    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03a9bc00-f973-4d50-ac32-a4ca843b977b-metrics-tls\") pod \"dns-default-5ztj2\" (UID: \"03a9bc00-f973-4d50-ac32-a4ca843b977b\") " pod="openshift-dns/dns-default-5ztj2"
Oct 30 16:46:54.849192 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:54.849157    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 16:46:54.849227 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:54.849216    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03a9bc00-f973-4d50-ac32-a4ca843b977b-metrics-tls podName:03a9bc00-f973-4d50-ac32-a4ca843b977b nodeName:}" failed. No retries permitted until 2026-10-30 16:46:55.849202817 +0000 UTC m=+32.631518061 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/03a9bc00-f973-4d50-ac32-a4ca843b977b-metrics-tls") pod "dns-default-5ztj2" (UID: "03a9bc00-f973-4d50-ac32-a4ca843b977b") : secret "dns-default-metrics-tls" not found
Oct 30 16:46:54.893591 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:54.893566    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ltvcq" event={"ID":"47bd3f72-e568-4ded-ae3f-398eceff7033","Type":"ContainerStarted","Data":"97014c2ed0aa27e2a90ced34214c7553866b01035fa740207826af713c3049ad"}
Oct 30 16:46:55.653353 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:55.653321    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:46:55.653632 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:55.653390    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2pxh"
Oct 30 16:46:55.653632 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:55.653390    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:46:55.672074 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:55.672038    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 16:46:55.672074 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:55.672047    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 16:46:55.672216 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:55.672109    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 16:46:55.672216 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:55.672168    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-szhrj\""
Oct 30 16:46:55.687179 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:55.687159    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 16:46:55.687294 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:55.687202    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9bdrw\""
Oct 30 16:46:55.687294 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:55.687234    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-8568q\""
Oct 30 16:46:55.855212 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:55.855177    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03a9bc00-f973-4d50-ac32-a4ca843b977b-metrics-tls\") pod \"dns-default-5ztj2\" (UID: \"03a9bc00-f973-4d50-ac32-a4ca843b977b\") " pod="openshift-dns/dns-default-5ztj2"
Oct 30 16:46:55.855689 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:55.855294    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 16:46:55.855689 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:55.855363    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03a9bc00-f973-4d50-ac32-a4ca843b977b-metrics-tls podName:03a9bc00-f973-4d50-ac32-a4ca843b977b nodeName:}" failed. No retries permitted until 2026-10-30 16:46:57.855345703 +0000 UTC m=+34.637660958 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/03a9bc00-f973-4d50-ac32-a4ca843b977b-metrics-tls") pod "dns-default-5ztj2" (UID: "03a9bc00-f973-4d50-ac32-a4ca843b977b") : secret "dns-default-metrics-tls" not found
Oct 30 16:46:57.366420 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:57.366376    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs\") pod \"network-metrics-daemon-7zf5f\" (UID: \"980827e6-3d1b-4924-acde-8d408d5e50df\") " pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:46:57.366791 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:57.366434    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/498ef897-364c-4e19-9fbf-cdc46aac623f-original-pull-secret\") pod \"global-pull-secret-syncer-5cgn8\" (UID: \"498ef897-364c-4e19-9fbf-cdc46aac623f\") " pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:46:57.366791 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:57.366531    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 16:46:57.366791 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:57.366598    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs podName:980827e6-3d1b-4924-acde-8d408d5e50df nodeName:}" failed. No retries permitted until 2026-10-30 16:47:29.366583494 +0000 UTC m=+66.148898735 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs") pod "network-metrics-daemon-7zf5f" (UID: "980827e6-3d1b-4924-acde-8d408d5e50df") : secret "metrics-daemon-secret" not found
Oct 30 16:46:57.368624 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:57.368606    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/498ef897-364c-4e19-9fbf-cdc46aac623f-original-pull-secret\") pod \"global-pull-secret-syncer-5cgn8\" (UID: \"498ef897-364c-4e19-9fbf-cdc46aac623f\") " pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:46:57.467237 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:57.467212    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhxl7\" (UniqueName: \"kubernetes.io/projected/7bfc195d-6fbc-45c8-9e2a-f9c206838651-kube-api-access-jhxl7\") pod \"network-check-target-r2pxh\" (UID: \"7bfc195d-6fbc-45c8-9e2a-f9c206838651\") " pod="openshift-network-diagnostics/network-check-target-r2pxh"
Oct 30 16:46:57.469452 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:57.469429    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhxl7\" (UniqueName: \"kubernetes.io/projected/7bfc195d-6fbc-45c8-9e2a-f9c206838651-kube-api-access-jhxl7\") pod \"network-check-target-r2pxh\" (UID: \"7bfc195d-6fbc-45c8-9e2a-f9c206838651\") " pod="openshift-network-diagnostics/network-check-target-r2pxh"
Oct 30 16:46:57.495437 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:57.495394    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r2pxh"
Oct 30 16:46:57.517672 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:57.517650    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:46:57.827287 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:57.827260    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r2pxh"]
Oct 30 16:46:57.829163 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:57.829138    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bfc195d_6fbc_45c8_9e2a_f9c206838651.slice/crio-e82e8a05438606ab56ebf1b3cd180cb4f419dcabb8450e9fbb8165427ea18c1d WatchSource:0}: Error finding container e82e8a05438606ab56ebf1b3cd180cb4f419dcabb8450e9fbb8165427ea18c1d: Status 404 returned error can't find the container with id e82e8a05438606ab56ebf1b3cd180cb4f419dcabb8450e9fbb8165427ea18c1d
Oct 30 16:46:57.850248 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:57.850227    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5cgn8"]
Oct 30 16:46:57.851762 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:57.851738    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod498ef897_364c_4e19_9fbf_cdc46aac623f.slice/crio-82bdbc4f6fde037dfb767e8ee70f82a0406ab4938445977074be2fef20f18ef7 WatchSource:0}: Error finding container 82bdbc4f6fde037dfb767e8ee70f82a0406ab4938445977074be2fef20f18ef7: Status 404 returned error can't find the container with id 82bdbc4f6fde037dfb767e8ee70f82a0406ab4938445977074be2fef20f18ef7
Oct 30 16:46:57.870786 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:57.870767    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03a9bc00-f973-4d50-ac32-a4ca843b977b-metrics-tls\") pod \"dns-default-5ztj2\" (UID: \"03a9bc00-f973-4d50-ac32-a4ca843b977b\") " pod="openshift-dns/dns-default-5ztj2"
Oct 30 16:46:57.870876 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:57.870863    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 16:46:57.870913 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:57.870905    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03a9bc00-f973-4d50-ac32-a4ca843b977b-metrics-tls podName:03a9bc00-f973-4d50-ac32-a4ca843b977b nodeName:}" failed. No retries permitted until 2026-10-30 16:47:01.870894048 +0000 UTC m=+38.653209302 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/03a9bc00-f973-4d50-ac32-a4ca843b977b-metrics-tls") pod "dns-default-5ztj2" (UID: "03a9bc00-f973-4d50-ac32-a4ca843b977b") : secret "dns-default-metrics-tls" not found
Oct 30 16:46:57.901590 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:57.901558    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ltvcq" event={"ID":"47bd3f72-e568-4ded-ae3f-398eceff7033","Type":"ContainerStarted","Data":"b16242e902039a768111580f8fa623ebdc3800c2afb5db0a19aac25a5ae3a09f"}
Oct 30 16:46:57.902686 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:57.902664    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5cgn8" event={"ID":"498ef897-364c-4e19-9fbf-cdc46aac623f","Type":"ContainerStarted","Data":"82bdbc4f6fde037dfb767e8ee70f82a0406ab4938445977074be2fef20f18ef7"}
Oct 30 16:46:57.903687 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:57.903670    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r2pxh" event={"ID":"7bfc195d-6fbc-45c8-9e2a-f9c206838651","Type":"ContainerStarted","Data":"e82e8a05438606ab56ebf1b3cd180cb4f419dcabb8450e9fbb8165427ea18c1d"}
Oct 30 16:46:57.948899 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:57.948828    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ltvcq" podStartSLOduration=1.297684427 podStartE2EDuration="3.948816s" totalImagesPullingTime="2.651131573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:57.948675473 +0000 UTC m=+34.730990756" watchObservedRunningTime="2026-10-30 16:46:57.948816 +0000 UTC m=+34.731131260"
Oct 30 16:46:58.319860 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:58.319828    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mrllk"]
Oct 30 16:46:58.340208 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:58.340181    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:46:58.369062 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:58.369032    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 16:46:58.378188 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:58.378162    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 16:46:58.378338 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:58.378278    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 16:46:58.379141 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:58.379121    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mrllk"]
Oct 30 16:46:58.425535 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:58.425503    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 16:46:58.425691 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:58.425566    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7stlv\""
Oct 30 16:46:58.475381 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:58.475350    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:46:58.475544 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:58.475413    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqjp6\" (UniqueName: \"kubernetes.io/projected/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-kube-api-access-vqjp6\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:46:58.475544 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:58.475519    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-crio-socket\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:46:58.475642 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:58.475597    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-data-volume\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:46:58.475642 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:58.475629    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:46:58.514261 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:58.508340    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-b46ww"]
Oct 30 16:46:58.525609 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:58.525575    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b46ww"
Oct 30 16:46:58.529794 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:58.529759    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-b46ww"]
Oct 30 16:46:58.570786 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:58.570696    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-vmkvz\""
Oct 30 16:46:58.574205 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:58.574182    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 16:46:58.574335 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:58.574245    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 16:46:58.576732 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:58.576707    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:46:58.576840 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:58.576756    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqjp6\" (UniqueName: \"kubernetes.io/projected/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-kube-api-access-vqjp6\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:46:58.576887 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:58.576854    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 16:46:58.576952 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:58.576939    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-kube-rbac-proxy-cm podName:0b4a144d-1016-44d7-88ac-0e7c8db3e3c3 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:59.07691872 +0000 UTC m=+35.859233963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mrllk" (UID: "0b4a144d-1016-44d7-88ac-0e7c8db3e3c3") : configmap "kube-rbac-proxy" not found
Oct 30 16:46:58.577067 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:58.577049    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-crio-socket\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:46:58.577121 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:58.577103    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-data-volume\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:46:58.577202 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:58.577143    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:46:58.577289 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:58.577272    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 16:46:58.577289 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:58.577280    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-crio-socket\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:46:58.577376 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:58.577321    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-insights-runtime-extractor-tls podName:0b4a144d-1016-44d7-88ac-0e7c8db3e3c3 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:59.077306711 +0000 UTC m=+35.859621957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mrllk" (UID: "0b4a144d-1016-44d7-88ac-0e7c8db3e3c3") : secret "insights-runtime-extractor-tls" not found
Oct 30 16:46:58.577611 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:58.577595    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-data-volume\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:46:58.605060 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:58.605035    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqjp6\" (UniqueName: \"kubernetes.io/projected/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-kube-api-access-vqjp6\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:46:58.677658 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:58.677617    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdnrp\" (UniqueName: \"kubernetes.io/projected/631feafb-af15-4230-a5d7-86357d9d3953-kube-api-access-cdnrp\") pod \"migrator-66df5d496b-b46ww\" (UID: \"631feafb-af15-4230-a5d7-86357d9d3953\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b46ww"
Oct 30 16:46:58.778230 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:58.778189    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdnrp\" (UniqueName: \"kubernetes.io/projected/631feafb-af15-4230-a5d7-86357d9d3953-kube-api-access-cdnrp\") pod \"migrator-66df5d496b-b46ww\" (UID: \"631feafb-af15-4230-a5d7-86357d9d3953\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b46ww"
Oct 30 16:46:58.811240 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:58.811209    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdnrp\" (UniqueName: \"kubernetes.io/projected/631feafb-af15-4230-a5d7-86357d9d3953-kube-api-access-cdnrp\") pod \"migrator-66df5d496b-b46ww\" (UID: \"631feafb-af15-4230-a5d7-86357d9d3953\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b46ww"
Oct 30 16:46:58.855139 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:58.855061    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b46ww"
Oct 30 16:46:59.080960 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:59.080729    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:46:59.080960 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:59.080808    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:46:59.080960 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:59.080867    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 16:46:59.080960 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:59.080905    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 16:46:59.080960 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:59.080945    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-kube-rbac-proxy-cm podName:0b4a144d-1016-44d7-88ac-0e7c8db3e3c3 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:00.080921249 +0000 UTC m=+36.863236504 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mrllk" (UID: "0b4a144d-1016-44d7-88ac-0e7c8db3e3c3") : configmap "kube-rbac-proxy" not found
Oct 30 16:46:59.080960 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:46:59.080964    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-insights-runtime-extractor-tls podName:0b4a144d-1016-44d7-88ac-0e7c8db3e3c3 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:00.080954775 +0000 UTC m=+36.863270022 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mrllk" (UID: "0b4a144d-1016-44d7-88ac-0e7c8db3e3c3") : secret "insights-runtime-extractor-tls" not found
Oct 30 16:46:59.175546 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:59.175435    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-b46ww"]
Oct 30 16:46:59.178020 ip-10-0-131-13 kubenswrapper[2581]: W1030 16:46:59.177974    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod631feafb_af15_4230_a5d7_86357d9d3953.slice/crio-a51865ae3b4dce0f9d9d528a37c4ec075132bcd9ef3d20d0a6dcdd87a786e3b6 WatchSource:0}: Error finding container a51865ae3b4dce0f9d9d528a37c4ec075132bcd9ef3d20d0a6dcdd87a786e3b6: Status 404 returned error can't find the container with id a51865ae3b4dce0f9d9d528a37c4ec075132bcd9ef3d20d0a6dcdd87a786e3b6
Oct 30 16:46:59.911060 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:46:59.910984    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b46ww" event={"ID":"631feafb-af15-4230-a5d7-86357d9d3953","Type":"ContainerStarted","Data":"a51865ae3b4dce0f9d9d528a37c4ec075132bcd9ef3d20d0a6dcdd87a786e3b6"}
Oct 30 16:47:00.087115 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:47:00.087076    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:47:00.087292 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:47:00.087154    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:47:00.087292 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:47:00.087249    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 16:47:00.087432 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:47:00.087308    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-kube-rbac-proxy-cm podName:0b4a144d-1016-44d7-88ac-0e7c8db3e3c3 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:02.087290855 +0000 UTC m=+38.869606096 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mrllk" (UID: "0b4a144d-1016-44d7-88ac-0e7c8db3e3c3") : configmap "kube-rbac-proxy" not found
Oct 30 16:47:00.087619 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:47:00.087593    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 16:47:00.087792 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:47:00.087664    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-insights-runtime-extractor-tls podName:0b4a144d-1016-44d7-88ac-0e7c8db3e3c3 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:02.087649083 +0000 UTC m=+38.869964328 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mrllk" (UID: "0b4a144d-1016-44d7-88ac-0e7c8db3e3c3") : secret "insights-runtime-extractor-tls" not found
Oct 30 16:47:01.902134 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:47:01.902040    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03a9bc00-f973-4d50-ac32-a4ca843b977b-metrics-tls\") pod \"dns-default-5ztj2\" (UID: \"03a9bc00-f973-4d50-ac32-a4ca843b977b\") " pod="openshift-dns/dns-default-5ztj2"
Oct 30 16:47:01.902637 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:47:01.902209    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 16:47:01.902637 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:47:01.902289    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03a9bc00-f973-4d50-ac32-a4ca843b977b-metrics-tls podName:03a9bc00-f973-4d50-ac32-a4ca843b977b nodeName:}" failed. No retries permitted until 2026-10-30 16:47:09.902268354 +0000 UTC m=+46.684583608 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/03a9bc00-f973-4d50-ac32-a4ca843b977b-metrics-tls") pod "dns-default-5ztj2" (UID: "03a9bc00-f973-4d50-ac32-a4ca843b977b") : secret "dns-default-metrics-tls" not found
Oct 30 16:47:01.917857 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:47:01.917822    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r2pxh" event={"ID":"7bfc195d-6fbc-45c8-9e2a-f9c206838651","Type":"ContainerStarted","Data":"6938a4e1b22ef7dc296292a42805f8bcf6df66a18354a7ad9cf506f80ffdaeb1"}
Oct 30 16:47:01.918021 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:47:01.917915    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-r2pxh"
Oct 30 16:47:01.919509 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:47:01.919486    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b46ww" event={"ID":"631feafb-af15-4230-a5d7-86357d9d3953","Type":"ContainerStarted","Data":"b6da95d6a1cf7954efcb280b7338cac8a0cbad3e2c46795a443133fc8c13e6aa"}
Oct 30 16:47:01.964852 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:47:01.964796    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-r2pxh" podStartSLOduration=34.196024298 podStartE2EDuration="37.964783494s" totalImagesPullingTime="3.768759196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:01.96404103 +0000 UTC m=+38.746356294" watchObservedRunningTime="2026-10-30 16:47:01.964783494 +0000 UTC m=+38.747098768"
Oct 30 16:47:02.009612 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:47:02.009551    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b46ww" podStartSLOduration=1.910130585 podStartE2EDuration="4.009534315s" totalImagesPullingTime="2.09940373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:02.009244553 +0000 UTC m=+38.791559815" watchObservedRunningTime="2026-10-30 16:47:02.009534315 +0000 UTC m=+38.791849577"
Oct 30 16:47:02.103789 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:47:02.103750    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:47:02.103981 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:47:02.103838    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:47:02.103981 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:47:02.103878    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 16:47:02.103981 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:47:02.103951    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-kube-rbac-proxy-cm podName:0b4a144d-1016-44d7-88ac-0e7c8db3e3c3 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:06.103933684 +0000 UTC m=+42.886248945 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mrllk" (UID: "0b4a144d-1016-44d7-88ac-0e7c8db3e3c3") : configmap "kube-rbac-proxy" not found
Oct 30 16:47:02.103981 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:47:02.103971    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 16:47:02.104160 ip-10-0-131-13 kubenswrapper[2581]: E1030 16:47:02.104005    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-insights-runtime-extractor-tls podName:0b4a144d-1016-44d7-88ac-0e7c8db3e3c3 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:06.103996065 +0000 UTC m=+42.886311310 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mrllk" (UID: "0b4a144d-1016-44d7-88ac-0e7c8db3e3c3") : secret "insights-runtime-extractor-tls" not found
Oct 30 16:47:03.502287 ip-10-0-131-13 kubenswrapper[2581]: I1030 16:47:03.502171    2581 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 16:47:03.502336 ip-10-0-131-13 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 16:47:03.515776 ip-10-0-131-13 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 16:47:03.515954 ip-10-0-131-13 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 16:47:03.516003 ip-10-0-131-13 systemd[1]: kubelet.service: Consumed 2.310s CPU time.
Oct 30 16:47:03.527980 ip-10-0-131-13 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 16:47:03.608388 ip-10-0-131-13 kubenswrapper[4433]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 16:47:03.608388 ip-10-0-131-13 kubenswrapper[4433]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 16:47:03.608388 ip-10-0-131-13 kubenswrapper[4433]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 16:47:03.608388 ip-10-0-131-13 kubenswrapper[4433]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 16:47:03.610509 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610419    4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 16:47:03.610509 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610507    4433 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 16:47:03.610509 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610511    4433 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 16:47:03.610621 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610514    4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 16:47:03.610621 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610518    4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 16:47:03.610621 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610520    4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 16:47:03.610621 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610523    4433 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 16:47:03.610621 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610528    4433 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 16:47:03.610621 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610534    4433 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 16:47:03.610621 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610538    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 16:47:03.610621 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610540    4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 16:47:03.610621 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610543    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 16:47:03.610621 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610546    4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 16:47:03.610621 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610549    4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 16:47:03.610621 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610552    4433 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 16:47:03.610621 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610555    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 16:47:03.610621 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610558    4433 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 16:47:03.610621 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610561    4433 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 16:47:03.610621 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610563    4433 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 16:47:03.610621 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610566    4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 16:47:03.610621 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610569    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 16:47:03.610621 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610571    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 16:47:03.611088 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610574    4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 16:47:03.611088 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610577    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 16:47:03.611088 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610580    4433 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 16:47:03.611088 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610583    4433 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 16:47:03.611088 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610586    4433 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 16:47:03.611088 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610589    4433 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 16:47:03.611088 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610591    4433 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 16:47:03.611088 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610594    4433 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 16:47:03.611088 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610597    4433 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 16:47:03.611088 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610600    4433 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 16:47:03.611088 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610602    4433 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 16:47:03.611088 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610605    4433 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 16:47:03.611088 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610608    4433 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 16:47:03.611088 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610612    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 16:47:03.611088 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610617    4433 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 16:47:03.611088 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610620    4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 16:47:03.611088 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610623    4433 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 16:47:03.611088 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610626    4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 16:47:03.611088 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610629    4433 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 16:47:03.611576 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610632    4433 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 16:47:03.611576 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610634    4433 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 16:47:03.611576 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610637    4433 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 16:47:03.611576 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610641    4433 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 16:47:03.611576 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610644    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 16:47:03.611576 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610646    4433 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 16:47:03.611576 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610649    4433 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 16:47:03.611576 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610652    4433 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 16:47:03.611576 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610655    4433 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 16:47:03.611576 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610657    4433 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 16:47:03.611576 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610661    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 16:47:03.611576 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610664    4433 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 16:47:03.611576 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610667    4433 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 16:47:03.611576 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610669    4433 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 16:47:03.611576 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610672    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 16:47:03.611576 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610675    4433 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 16:47:03.611576 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610678    4433 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 16:47:03.611576 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610680    4433 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 16:47:03.611576 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610683    4433 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 16:47:03.612059 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610686    4433 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 16:47:03.612059 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610688    4433 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 16:47:03.612059 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610691    4433 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 16:47:03.612059 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610693    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 16:47:03.612059 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610696    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 16:47:03.612059 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610699    4433 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 16:47:03.612059 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610702    4433 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 16:47:03.612059 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610705    4433 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 16:47:03.612059 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610708    4433 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 16:47:03.612059 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610711    4433 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 16:47:03.612059 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610714    4433 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 16:47:03.612059 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610716    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 16:47:03.612059 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610719    4433 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 16:47:03.612059 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610722    4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 16:47:03.612059 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610724    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 16:47:03.612059 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610727    4433 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 16:47:03.612059 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610729    4433 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 16:47:03.612059 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610732    4433 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 16:47:03.612059 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610735    4433 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 16:47:03.612059 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610737    4433 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 16:47:03.612556 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610740    4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 16:47:03.612556 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610744    4433 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 16:47:03.612556 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610746    4433 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 16:47:03.612556 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610749    4433 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 16:47:03.612556 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610752    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 16:47:03.612556 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610754    4433 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 16:47:03.612556 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610757    4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 16:47:03.612556 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610760    4433 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 16:47:03.612556 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610762    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 16:47:03.612556 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610765    4433 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 16:47:03.612556 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610767    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 16:47:03.612556 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610770    4433 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 16:47:03.612556 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610772    4433 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 16:47:03.612556 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610775    4433 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 16:47:03.612556 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610778    4433 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 16:47:03.612556 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610780    4433 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 16:47:03.612556 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610783    4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 16:47:03.612556 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610786    4433 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 16:47:03.612556 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610790    4433 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 16:47:03.612556 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610792    4433 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 16:47:03.613058 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610795    4433 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 16:47:03.613058 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610798    4433 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 16:47:03.613058 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610800    4433 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 16:47:03.613058 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610803    4433 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 16:47:03.613058 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610806    4433 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 16:47:03.613058 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.610808    4433 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 16:47:03.613058 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611647    4433 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 16:47:03.613058 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611657    4433 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 16:47:03.613058 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611661    4433 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 16:47:03.613058 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611665    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 16:47:03.613058 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611668    4433 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 16:47:03.613058 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611670    4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 16:47:03.613058 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611673    4433 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 16:47:03.613058 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611676    4433 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 16:47:03.613058 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611679    4433 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 16:47:03.613058 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611682    4433 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 16:47:03.613058 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611684    4433 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 16:47:03.613058 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611687    4433 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 16:47:03.613058 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611690    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 16:47:03.613058 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611693    4433 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 16:47:03.613058 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611695    4433 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 16:47:03.613581 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611698    4433 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 16:47:03.613581 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611701    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 16:47:03.613581 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611703    4433 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 16:47:03.613581 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611706    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 16:47:03.613581 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611708    4433 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 16:47:03.613581 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611711    4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 16:47:03.613581 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611714    4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 16:47:03.613581 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611717    4433 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 16:47:03.613581 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611720    4433 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 16:47:03.613581 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611722    4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 16:47:03.613581 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611725    4433 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 16:47:03.613581 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611728    4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 16:47:03.613581 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611732    4433 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 16:47:03.613581 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611735    4433 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 16:47:03.613581 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611737    4433 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 16:47:03.613581 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611740    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 16:47:03.613581 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611742    4433 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 16:47:03.613581 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611745    4433 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 16:47:03.613581 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611749    4433 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 16:47:03.614046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611753    4433 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 16:47:03.614046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611755    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 16:47:03.614046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611758    4433 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 16:47:03.614046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611761    4433 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 16:47:03.614046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611764    4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 16:47:03.614046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611766    4433 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 16:47:03.614046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611769    4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 16:47:03.614046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611771    4433 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 16:47:03.614046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611774    4433 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 16:47:03.614046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611777    4433 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 16:47:03.614046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611780    4433 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 16:47:03.614046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611782    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 16:47:03.614046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611785    4433 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 16:47:03.614046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611787    4433 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 16:47:03.614046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611790    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 16:47:03.614046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611793    4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 16:47:03.614046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611796    4433 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 16:47:03.614046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611798    4433 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 16:47:03.614046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611801    4433 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 16:47:03.614523 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611803    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 16:47:03.614523 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611806    4433 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 16:47:03.614523 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611808    4433 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 16:47:03.614523 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611811    4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 16:47:03.614523 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611820    4433 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 16:47:03.614523 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611826    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 16:47:03.614523 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611828    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 16:47:03.614523 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611831    4433 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 16:47:03.614523 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611834    4433 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 16:47:03.614523 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611836    4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 16:47:03.614523 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611839    4433 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 16:47:03.614523 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611841    4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 16:47:03.614523 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611845    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 16:47:03.614523 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611847    4433 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 16:47:03.614523 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611850    4433 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 16:47:03.614523 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611852    4433 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 16:47:03.614523 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611855    4433 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 16:47:03.614523 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611858    4433 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 16:47:03.614523 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611860    4433 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 16:47:03.615025 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611863    4433 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 16:47:03.615025 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611865    4433 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 16:47:03.615025 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611868    4433 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 16:47:03.615025 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611870    4433 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 16:47:03.615025 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611873    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 16:47:03.615025 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611876    4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 16:47:03.615025 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611878    4433 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 16:47:03.615025 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611881    4433 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 16:47:03.615025 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611885    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 16:47:03.615025 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611887    4433 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 16:47:03.615025 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611890    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 16:47:03.615025 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611892    4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 16:47:03.615025 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611895    4433 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 16:47:03.615025 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611898    4433 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 16:47:03.615025 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611900    4433 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 16:47:03.615025 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611903    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 16:47:03.615025 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611905    4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 16:47:03.615025 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611908    4433 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 16:47:03.615025 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611915    4433 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 16:47:03.615542 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611920    4433 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 16:47:03.615542 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611923    4433 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 16:47:03.615542 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611926    4433 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 16:47:03.615542 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611929    4433 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 16:47:03.615542 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611932    4433 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 16:47:03.615542 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611935    4433 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 16:47:03.615542 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611937    4433 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 16:47:03.615542 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611940    4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 16:47:03.615542 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611943    4433 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 16:47:03.615542 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611945    4433 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 16:47:03.615542 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611948    4433 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 16:47:03.615542 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611951    4433 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 16:47:03.615542 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611955    4433 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 16:47:03.615542 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611972    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 16:47:03.615542 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.611976    4433 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 16:47:03.615542 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612480    4433 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 16:47:03.615542 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612486    4433 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 16:47:03.615542 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612491    4433 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 16:47:03.615542 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612494    4433 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 16:47:03.616012 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612497    4433 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 16:47:03.616012 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612500    4433 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 16:47:03.616012 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612504    4433 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 16:47:03.616012 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612506    4433 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 16:47:03.616012 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612509    4433 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 16:47:03.616012 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612512    4433 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 16:47:03.616012 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612515    4433 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 16:47:03.616012 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612518    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 16:47:03.616012 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612520    4433 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 16:47:03.616012 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612523    4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 16:47:03.616012 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612526    4433 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 16:47:03.616012 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612528    4433 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 16:47:03.616012 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612533    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 16:47:03.616012 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612536    4433 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 16:47:03.616012 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612539    4433 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 16:47:03.616012 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612541    4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 16:47:03.616012 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612544    4433 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 16:47:03.616012 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612546    4433 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 16:47:03.616012 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612549    4433 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 16:47:03.616012 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612551    4433 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 16:47:03.616547 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612554    4433 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 16:47:03.616547 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612557    4433 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 16:47:03.616547 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612560    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 16:47:03.616547 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612563    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 16:47:03.616547 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612566    4433 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 16:47:03.616547 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612569    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 16:47:03.616547 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612572    4433 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 16:47:03.616547 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612575    4433 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 16:47:03.616547 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612577    4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 16:47:03.616547 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612580    4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 16:47:03.616547 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612582    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 16:47:03.616547 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612585    4433 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 16:47:03.616547 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612588    4433 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 16:47:03.616547 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612590    4433 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 16:47:03.616547 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612594    4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 16:47:03.616547 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612596    4433 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 16:47:03.616547 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612599    4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 16:47:03.616547 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612602    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 16:47:03.616547 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612604    4433 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 16:47:03.617135 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612608    4433 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 16:47:03.617135 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612612    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 16:47:03.617135 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612615    4433 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 16:47:03.617135 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612619    4433 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 16:47:03.617135 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612622    4433 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 16:47:03.617135 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612626    4433 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 16:47:03.617135 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612628    4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 16:47:03.617135 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612631    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 16:47:03.617135 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612634    4433 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 16:47:03.617135 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612637    4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 16:47:03.617135 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612639    4433 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 16:47:03.617135 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612643    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 16:47:03.617135 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612645    4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 16:47:03.617135 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612648    4433 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 16:47:03.617135 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612651    4433 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 16:47:03.617135 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612653    4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 16:47:03.617135 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612660    4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 16:47:03.617135 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612662    4433 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 16:47:03.617135 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612665    4433 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 16:47:03.617891 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612667    4433 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 16:47:03.617891 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612670    4433 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 16:47:03.617891 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612673    4433 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 16:47:03.617891 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612675    4433 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 16:47:03.617891 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612678    4433 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 16:47:03.617891 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612680    4433 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 16:47:03.617891 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612683    4433 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 16:47:03.617891 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612686    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 16:47:03.617891 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612688    4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 16:47:03.617891 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612691    4433 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 16:47:03.617891 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612694    4433 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 16:47:03.617891 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612696    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 16:47:03.617891 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612699    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 16:47:03.617891 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612701    4433 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 16:47:03.617891 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612704    4433 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 16:47:03.617891 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612706    4433 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 16:47:03.617891 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612709    4433 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 16:47:03.617891 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612712    4433 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 16:47:03.617891 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612714    4433 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 16:47:03.617891 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612717    4433 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 16:47:03.618467 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612719    4433 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 16:47:03.618467 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612722    4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 16:47:03.618467 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612725    4433 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 16:47:03.618467 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612729    4433 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 16:47:03.618467 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612731    4433 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 16:47:03.618467 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612734    4433 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 16:47:03.618467 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612736    4433 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 16:47:03.618467 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612739    4433 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 16:47:03.618467 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612742    4433 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 16:47:03.618467 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612746    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 16:47:03.618467 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612748    4433 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 16:47:03.618467 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612751    4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 16:47:03.618467 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612753    4433 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 16:47:03.618467 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612756    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 16:47:03.618467 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612759    4433 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 16:47:03.618467 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612761    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 16:47:03.618467 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612763    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 16:47:03.618467 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612766    4433 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 16:47:03.618467 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612769    4433 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 16:47:03.618937 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612771    4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 16:47:03.618937 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612774    4433 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 16:47:03.618937 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612776    4433 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 16:47:03.618937 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612779    4433 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 16:47:03.618937 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.612781    4433 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 16:47:03.618937 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.612881    4433 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 16:47:03.618937 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.612891    4433 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 16:47:03.618937 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.612898    4433 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 16:47:03.618937 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.612903    4433 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 16:47:03.618937 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.612908    4433 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 16:47:03.618937 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.612911    4433 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 16:47:03.618937 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.612915    4433 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 16:47:03.618937 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.612919    4433 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 16:47:03.618937 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.612923    4433 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 16:47:03.618937 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.612926    4433 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 16:47:03.618937 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.612931    4433 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 16:47:03.618937 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.612934    4433 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 16:47:03.618937 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.612937    4433 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 16:47:03.618937 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.612940    4433 flags.go:64] FLAG: --cgroup-root=""
Oct 30 16:47:03.618937 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.612943    4433 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 16:47:03.618937 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.612946    4433 flags.go:64] FLAG: --client-ca-file=""
Oct 30 16:47:03.618937 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.612949    4433 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 16:47:03.618937 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.612954    4433 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 16:47:03.619499 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.612959    4433 flags.go:64] FLAG: --cluster-domain=""
Oct 30 16:47:03.619499 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.612962    4433 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 16:47:03.619499 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.612965    4433 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 16:47:03.619499 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.612968    4433 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 16:47:03.619499 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.612972    4433 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 16:47:03.619499 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.612976    4433 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 16:47:03.619499 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.612979    4433 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 16:47:03.619499 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.612982    4433 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 16:47:03.619499 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.612985    4433 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 16:47:03.619499 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.612988    4433 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 16:47:03.619499 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.612992    4433 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 16:47:03.619499 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.612995    4433 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 16:47:03.619499 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.612998    4433 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 16:47:03.619499 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613001    4433 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 16:47:03.619499 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613005    4433 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 16:47:03.619499 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613009    4433 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 16:47:03.619499 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613012    4433 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 16:47:03.619499 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613015    4433 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 16:47:03.619499 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613018    4433 flags.go:64] FLAG: --enable-server="true"
Oct 30 16:47:03.619499 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613022    4433 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 16:47:03.619499 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613028    4433 flags.go:64] FLAG: --event-burst="100"
Oct 30 16:47:03.619499 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613031    4433 flags.go:64] FLAG: --event-qps="50"
Oct 30 16:47:03.619499 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613034    4433 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 16:47:03.619499 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613037    4433 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 16:47:03.620092 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613040    4433 flags.go:64] FLAG: --eviction-hard=""
Oct 30 16:47:03.620092 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613045    4433 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 16:47:03.620092 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613048    4433 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 16:47:03.620092 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613052    4433 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 16:47:03.620092 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613055    4433 flags.go:64] FLAG: --eviction-soft=""
Oct 30 16:47:03.620092 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613058    4433 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 16:47:03.620092 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613062    4433 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 16:47:03.620092 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613066    4433 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 16:47:03.620092 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613069    4433 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 16:47:03.620092 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613072    4433 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 16:47:03.620092 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613075    4433 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 16:47:03.620092 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613078    4433 flags.go:64] FLAG: --feature-gates=""
Oct 30 16:47:03.620092 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613085    4433 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 16:47:03.620092 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613088    4433 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 16:47:03.620092 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613092    4433 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 16:47:03.620092 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613095    4433 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 16:47:03.620092 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613098    4433 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 16:47:03.620092 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613101    4433 flags.go:64] FLAG: --help="false"
Oct 30 16:47:03.620092 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613104    4433 flags.go:64] FLAG: --hostname-override="ip-10-0-131-13.ec2.internal"
Oct 30 16:47:03.620092 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613108    4433 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 16:47:03.620092 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613111    4433 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 16:47:03.620092 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613114    4433 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 16:47:03.620092 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613117    4433 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 16:47:03.620092 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613121    4433 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 16:47:03.620716 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613124    4433 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 16:47:03.620716 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613127    4433 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 16:47:03.620716 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613130    4433 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 16:47:03.620716 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613133    4433 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 16:47:03.620716 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613137    4433 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 16:47:03.620716 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613141    4433 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 16:47:03.620716 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613144    4433 flags.go:64] FLAG: --kube-reserved=""
Oct 30 16:47:03.620716 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613147    4433 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 16:47:03.620716 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613150    4433 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 16:47:03.620716 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613153    4433 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 16:47:03.620716 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613156    4433 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 16:47:03.620716 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613159    4433 flags.go:64] FLAG: --lock-file=""
Oct 30 16:47:03.620716 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613162    4433 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 16:47:03.620716 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613165    4433 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 16:47:03.620716 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613168    4433 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 16:47:03.620716 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613175    4433 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 16:47:03.620716 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613178    4433 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 16:47:03.620716 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613181    4433 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 16:47:03.620716 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613184    4433 flags.go:64] FLAG: --logging-format="text"
Oct 30 16:47:03.620716 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613187    4433 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 16:47:03.620716 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613190    4433 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 16:47:03.620716 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613193    4433 flags.go:64] FLAG: --manifest-url=""
Oct 30 16:47:03.620716 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613196    4433 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 16:47:03.620716 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613201    4433 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 16:47:03.620716 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613204    4433 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 16:47:03.621311 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613208    4433 flags.go:64] FLAG: --max-pods="110"
Oct 30 16:47:03.621311 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613211    4433 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 16:47:03.621311 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613215    4433 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 16:47:03.621311 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613218    4433 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 16:47:03.621311 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613221    4433 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 16:47:03.621311 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613224    4433 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 16:47:03.621311 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613226    4433 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 16:47:03.621311 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613230    4433 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 16:47:03.621311 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613236    4433 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 16:47:03.621311 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613239    4433 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 16:47:03.621311 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613242    4433 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 16:47:03.621311 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613247    4433 flags.go:64] FLAG: --pod-cidr=""
Oct 30 16:47:03.621311 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613250    4433 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 16:47:03.621311 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613253    4433 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 16:47:03.621311 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613256    4433 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 16:47:03.621311 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613259    4433 flags.go:64] FLAG: --port="10250"
Oct 30 16:47:03.621311 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613262    4433 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 16:47:03.621311 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613265    4433 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-047c5dfc4d5a57b81"
Oct 30 16:47:03.621311 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613269    4433 flags.go:64] FLAG: --qos-reserved=""
Oct 30 16:47:03.621311 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613272    4433 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 16:47:03.621311 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613275    4433 flags.go:64] FLAG: --register-node="true"
Oct 30 16:47:03.621311 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613278    4433 flags.go:64] FLAG: --register-with-taints=""
Oct 30 16:47:03.621311 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613283    4433 flags.go:64] FLAG: --registry-burst="10"
Oct 30 16:47:03.621311 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613286    4433 flags.go:64] FLAG: --registry-qps="5"
Oct 30 16:47:03.621311 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613289    4433 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 16:47:03.621311 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613292    4433 flags.go:64] FLAG: --reserved-memory=""
Oct 30 16:47:03.622109 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613296    4433 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 16:47:03.622109 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613299    4433 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 16:47:03.622109 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613302    4433 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 16:47:03.622109 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613305    4433 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 16:47:03.622109 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613308    4433 flags.go:64] FLAG: --runonce="false"
Oct 30 16:47:03.622109 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613310    4433 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 16:47:03.622109 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613313    4433 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 16:47:03.622109 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613316    4433 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 16:47:03.622109 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613319    4433 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 16:47:03.622109 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613322    4433 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 16:47:03.622109 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613325    4433 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 16:47:03.622109 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613329    4433 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 16:47:03.622109 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613332    4433 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 16:47:03.622109 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613335    4433 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 16:47:03.622109 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613337    4433 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 16:47:03.622109 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613340    4433 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 16:47:03.622109 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613343    4433 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 16:47:03.622109 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613348    4433 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 16:47:03.622109 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613351    4433 flags.go:64] FLAG: --system-cgroups=""
Oct 30 16:47:03.622109 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613354    4433 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 16:47:03.622109 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613360    4433 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 16:47:03.622109 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613363    4433 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 16:47:03.622109 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613366    4433 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 16:47:03.622109 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613371    4433 flags.go:64] FLAG: --tls-min-version=""
Oct 30 16:47:03.622109 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613374    4433 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 16:47:03.622905 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613377    4433 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 16:47:03.622905 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613380    4433 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 16:47:03.622905 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613383    4433 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 16:47:03.622905 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613387    4433 flags.go:64] FLAG: --v="2"
Oct 30 16:47:03.622905 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613392    4433 flags.go:64] FLAG: --version="false"
Oct 30 16:47:03.622905 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613396    4433 flags.go:64] FLAG: --vmodule=""
Oct 30 16:47:03.622905 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613413    4433 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 16:47:03.622905 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613418    4433 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 16:47:03.622905 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613554    4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 16:47:03.622905 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613558    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 16:47:03.622905 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613561    4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 16:47:03.622905 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613564    4433 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 16:47:03.622905 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613567    4433 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 16:47:03.622905 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613570    4433 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 16:47:03.622905 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613572    4433 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 16:47:03.622905 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613575    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 16:47:03.622905 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613578    4433 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 16:47:03.622905 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613583    4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 16:47:03.622905 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613586    4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 16:47:03.622905 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613589    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 16:47:03.622905 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613591    4433 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 16:47:03.623470 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613595    4433 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 16:47:03.623470 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613597    4433 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 16:47:03.623470 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613600    4433 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 16:47:03.623470 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613604    4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 16:47:03.623470 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613606    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 16:47:03.623470 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613609    4433 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 16:47:03.623470 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613611    4433 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 16:47:03.623470 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613614    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 16:47:03.623470 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613616    4433 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 16:47:03.623470 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613619    4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 16:47:03.623470 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613622    4433 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 16:47:03.623470 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613624    4433 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 16:47:03.623470 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613627    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 16:47:03.623470 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613630    4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 16:47:03.623470 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613633    4433 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 16:47:03.623470 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613636    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 16:47:03.623470 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613640    4433 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 16:47:03.623470 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613644    4433 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 16:47:03.623470 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613648    4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 16:47:03.624046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613651    4433 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 16:47:03.624046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613654    4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 16:47:03.624046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613656    4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 16:47:03.624046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613659    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 16:47:03.624046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613661    4433 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 16:47:03.624046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613664    4433 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 16:47:03.624046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613667    4433 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 16:47:03.624046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613669    4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 16:47:03.624046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613671    4433 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 16:47:03.624046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613674    4433 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 16:47:03.624046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613677    4433 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 16:47:03.624046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613679    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 16:47:03.624046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613682    4433 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 16:47:03.624046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613684    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 16:47:03.624046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613687    4433 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 16:47:03.624046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613690    4433 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 16:47:03.624046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613693    4433 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 16:47:03.624046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613696    4433 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 16:47:03.624046 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613698    4433 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 16:47:03.624570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613701    4433 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 16:47:03.624570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613703    4433 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 16:47:03.624570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613706    4433 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 16:47:03.624570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613709    4433 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 16:47:03.624570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613712    4433 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 16:47:03.624570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613714    4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 16:47:03.624570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613717    4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 16:47:03.624570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613720    4433 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 16:47:03.624570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613723    4433 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 16:47:03.624570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613726    4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 16:47:03.624570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613728    4433 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 16:47:03.624570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613731    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 16:47:03.624570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613733    4433 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 16:47:03.624570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613736    4433 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 16:47:03.624570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613738    4433 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 16:47:03.624570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613741    4433 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 16:47:03.624570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613744    4433 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 16:47:03.624570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613746    4433 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 16:47:03.624570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613749    4433 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 16:47:03.624570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613751    4433 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 16:47:03.625224 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613754    4433 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 16:47:03.625224 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613756    4433 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 16:47:03.625224 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613759    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 16:47:03.625224 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613761    4433 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 16:47:03.625224 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613764    4433 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 16:47:03.625224 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613766    4433 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 16:47:03.625224 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613769    4433 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 16:47:03.625224 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613775    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 16:47:03.625224 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613778    4433 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 16:47:03.625224 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613783    4433 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 16:47:03.625224 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613785    4433 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 16:47:03.625224 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613788    4433 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 16:47:03.625224 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613790    4433 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 16:47:03.625224 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613793    4433 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 16:47:03.625224 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613795    4433 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 16:47:03.625224 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613798    4433 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 16:47:03.625224 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613801    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 16:47:03.625224 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613803    4433 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 16:47:03.625224 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613806    4433 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 16:47:03.625224 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613808    4433 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 16:47:03.625788 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613811    4433 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 16:47:03.625788 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613814    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 16:47:03.625788 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613816    4433 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 16:47:03.625788 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613819    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 16:47:03.625788 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613822    4433 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 16:47:03.625788 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613825    4433 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 16:47:03.625788 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613827    4433 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 16:47:03.625788 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613830    4433 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 16:47:03.625788 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613832    4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 16:47:03.625788 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613835    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 16:47:03.625788 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613837    4433 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 16:47:03.625788 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613840    4433 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 16:47:03.625788 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613843    4433 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 16:47:03.625788 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613847    4433 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 16:47:03.625788 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613851    4433 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 16:47:03.625788 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.613877    4433 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 16:47:03.626200 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613913    4433 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 16:47:03.626200 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613918    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 16:47:03.626200 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613922    4433 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 16:47:03.626200 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613925    4433 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 16:47:03.626200 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613927    4433 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 16:47:03.626200 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613931    4433 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 16:47:03.626200 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613934    4433 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 16:47:03.626200 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613937    4433 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 16:47:03.626200 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613939    4433 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 16:47:03.626200 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613942    4433 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 16:47:03.626200 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613944    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 16:47:03.626200 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613947    4433 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 16:47:03.626200 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613950    4433 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 16:47:03.626200 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613952    4433 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 16:47:03.626200 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613955    4433 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 16:47:03.626200 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613957    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 16:47:03.626200 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613960    4433 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 16:47:03.626200 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613963    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 16:47:03.626200 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613966    4433 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 16:47:03.626200 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613969    4433 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 16:47:03.626761 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613972    4433 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 16:47:03.626761 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613974    4433 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 16:47:03.626761 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613977    4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 16:47:03.626761 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613979    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 16:47:03.626761 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613982    4433 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 16:47:03.626761 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613984    4433 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 16:47:03.626761 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613988    4433 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 16:47:03.626761 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613992    4433 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 16:47:03.626761 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613995    4433 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 16:47:03.626761 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.613999    4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 16:47:03.626761 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614002    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 16:47:03.626761 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614004    4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 16:47:03.626761 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614007    4433 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 16:47:03.626761 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614010    4433 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 16:47:03.626761 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614012    4433 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 16:47:03.626761 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614015    4433 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 16:47:03.626761 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614018    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 16:47:03.626761 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614021    4433 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 16:47:03.626761 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614024    4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 16:47:03.627336 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614026    4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 16:47:03.627336 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614029    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 16:47:03.627336 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614031    4433 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 16:47:03.627336 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614034    4433 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 16:47:03.627336 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614038    4433 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 16:47:03.627336 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614041    4433 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 16:47:03.627336 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614044    4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 16:47:03.627336 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614046    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 16:47:03.627336 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614049    4433 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 16:47:03.627336 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614052    4433 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 16:47:03.627336 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614055    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 16:47:03.627336 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614058    4433 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 16:47:03.627336 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614061    4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 16:47:03.627336 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614064    4433 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 16:47:03.627336 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614066    4433 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 16:47:03.627336 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614069    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 16:47:03.627336 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614071    4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 16:47:03.627336 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614074    4433 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 16:47:03.627336 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614076    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 16:47:03.627987 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614080    4433 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 16:47:03.627987 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614084    4433 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 16:47:03.627987 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614087    4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 16:47:03.627987 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614089    4433 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 16:47:03.627987 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614093    4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 16:47:03.627987 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614096    4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 16:47:03.627987 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614098    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 16:47:03.627987 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614101    4433 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 16:47:03.627987 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614104    4433 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 16:47:03.627987 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614106    4433 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 16:47:03.627987 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614109    4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 16:47:03.627987 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614113    4433 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 16:47:03.627987 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614115    4433 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 16:47:03.627987 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614118    4433 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 16:47:03.627987 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614120    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 16:47:03.627987 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614123    4433 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 16:47:03.627987 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614125    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 16:47:03.627987 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614129    4433 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 16:47:03.627987 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614132    4433 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 16:47:03.628783 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614135    4433 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 16:47:03.628783 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614137    4433 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 16:47:03.628783 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614140    4433 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 16:47:03.628783 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614143    4433 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 16:47:03.628783 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614145    4433 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 16:47:03.628783 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614148    4433 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 16:47:03.628783 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614151    4433 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 16:47:03.628783 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614154    4433 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 16:47:03.628783 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614157    4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 16:47:03.628783 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614159    4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 16:47:03.628783 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614162    4433 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 16:47:03.628783 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614164    4433 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 16:47:03.628783 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614167    4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 16:47:03.628783 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614169    4433 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 16:47:03.628783 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614172    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 16:47:03.628783 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614175    4433 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 16:47:03.628783 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614177    4433 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 16:47:03.628783 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614180    4433 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 16:47:03.628783 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614184    4433 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 16:47:03.628783 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614186    4433 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 16:47:03.629618 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614189    4433 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 16:47:03.629618 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614191    4433 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 16:47:03.629618 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614194    4433 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 16:47:03.629618 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614196    4433 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 16:47:03.629618 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614200    4433 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 16:47:03.629618 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614203    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 16:47:03.629618 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614205    4433 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 16:47:03.629618 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614208    4433 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 16:47:03.629618 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.614210    4433 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 16:47:03.629618 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.619555    4433 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 16:47:03.629618 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.619572    4433 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 16:47:03.629618 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.619597    4433 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 16:47:03.629618 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.619602    4433 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 16:47:03.629618 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619663    4433 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 16:47:03.629618 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619669    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 16:47:03.629618 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619673    4433 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 16:47:03.629618 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619676    4433 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 16:47:03.629618 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619679    4433 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 16:47:03.629618 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619681    4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 16:47:03.630463 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619684    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 16:47:03.630463 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619687    4433 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 16:47:03.630463 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619690    4433 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 16:47:03.630463 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619692    4433 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 16:47:03.630463 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619695    4433 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 16:47:03.630463 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619698    4433 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 16:47:03.630463 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619701    4433 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 16:47:03.630463 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619703    4433 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 16:47:03.630463 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619706    4433 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 16:47:03.630463 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619709    4433 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 16:47:03.630463 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619711    4433 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 16:47:03.630463 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619714    4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 16:47:03.630463 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619717    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 16:47:03.630463 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619720    4433 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 16:47:03.630463 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619723    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 16:47:03.630463 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619726    4433 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 16:47:03.630463 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619729    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 16:47:03.630463 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619732    4433 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 16:47:03.630463 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619736    4433 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 16:47:03.630463 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619740    4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 16:47:03.631261 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619743    4433 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 16:47:03.631261 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619746    4433 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 16:47:03.631261 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619749    4433 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 16:47:03.631261 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619752    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 16:47:03.631261 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619755    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 16:47:03.631261 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619758    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 16:47:03.631261 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619760    4433 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 16:47:03.631261 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619763    4433 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 16:47:03.631261 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619766    4433 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 16:47:03.631261 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619768    4433 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 16:47:03.631261 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619771    4433 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 16:47:03.631261 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619774    4433 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 16:47:03.631261 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619776    4433 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 16:47:03.631261 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619779    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 16:47:03.631261 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619782    4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 16:47:03.631261 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619784    4433 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 16:47:03.631261 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619787    4433 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 16:47:03.631261 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619790    4433 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 16:47:03.631261 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619792    4433 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 16:47:03.631261 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619795    4433 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 16:47:03.632351 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619805    4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 16:47:03.632351 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619808    4433 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 16:47:03.632351 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619811    4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 16:47:03.632351 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619822    4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 16:47:03.632351 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619826    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 16:47:03.632351 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619829    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 16:47:03.632351 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619832    4433 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 16:47:03.632351 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619834    4433 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 16:47:03.632351 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619837    4433 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 16:47:03.632351 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619840    4433 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 16:47:03.632351 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619842    4433 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 16:47:03.632351 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619845    4433 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 16:47:03.632351 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619848    4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 16:47:03.632351 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619851    4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 16:47:03.632351 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619853    4433 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 16:47:03.632351 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619856    4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 16:47:03.632351 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619858    4433 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 16:47:03.632351 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619861    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 16:47:03.632351 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619864    4433 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 16:47:03.632351 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619866    4433 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 16:47:03.633213 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619869    4433 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 16:47:03.633213 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619871    4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 16:47:03.633213 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619874    4433 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 16:47:03.633213 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619876    4433 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 16:47:03.633213 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619879    4433 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 16:47:03.633213 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619882    4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 16:47:03.633213 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619884    4433 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 16:47:03.633213 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619887    4433 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 16:47:03.633213 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619889    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 16:47:03.633213 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619892    4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 16:47:03.633213 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619894    4433 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 16:47:03.633213 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619897    4433 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 16:47:03.633213 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619900    4433 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 16:47:03.633213 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619904    4433 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 16:47:03.633213 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619907    4433 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 16:47:03.633213 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619910    4433 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 16:47:03.633213 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619913    4433 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 16:47:03.633213 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619915    4433 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 16:47:03.633213 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619918    4433 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 16:47:03.633213 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619921    4433 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 16:47:03.634010 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619923    4433 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 16:47:03.634010 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619926    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 16:47:03.634010 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619928    4433 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 16:47:03.634010 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619931    4433 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 16:47:03.634010 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619933    4433 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 16:47:03.634010 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619936    4433 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 16:47:03.634010 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619939    4433 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 16:47:03.634010 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619941    4433 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 16:47:03.634010 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619945    4433 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 16:47:03.634010 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619949    4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 16:47:03.634010 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619952    4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 16:47:03.634010 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619955    4433 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 16:47:03.634010 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619958    4433 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 16:47:03.634010 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619961    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 16:47:03.634010 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619964    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 16:47:03.634010 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619967    4433 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 16:47:03.634010 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619969    4433 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 16:47:03.634010 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619972    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 16:47:03.634010 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619974    4433 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 16:47:03.634570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.619977    4433 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 16:47:03.634570 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.620004    4433 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 16:47:03.634570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620144    4433 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 16:47:03.634570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620150    4433 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 16:47:03.634570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620153    4433 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 16:47:03.634570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620156    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 16:47:03.634570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620159    4433 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 16:47:03.634570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620163    4433 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 16:47:03.634570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620165    4433 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 16:47:03.634570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620169    4433 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 16:47:03.634570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620172    4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 16:47:03.634570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620174    4433 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 16:47:03.634570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620177    4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 16:47:03.634570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620180    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 16:47:03.634570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620182    4433 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 16:47:03.634570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620185    4433 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 16:47:03.634970 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620188    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 16:47:03.634970 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620190    4433 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 16:47:03.634970 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620193    4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 16:47:03.634970 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620196    4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 16:47:03.634970 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620198    4433 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 16:47:03.634970 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620201    4433 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 16:47:03.634970 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620204    4433 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 16:47:03.634970 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620206    4433 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 16:47:03.634970 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620209    4433 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 16:47:03.634970 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620212    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 16:47:03.634970 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620215    4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 16:47:03.634970 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620218    4433 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 16:47:03.634970 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620221    4433 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 16:47:03.634970 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620223    4433 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 16:47:03.634970 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620226    4433 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 16:47:03.634970 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620228    4433 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 16:47:03.634970 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620231    4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 16:47:03.634970 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620233    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 16:47:03.634970 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620236    4433 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 16:47:03.634970 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620238    4433 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 16:47:03.635481 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620241    4433 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 16:47:03.635481 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620245    4433 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 16:47:03.635481 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620249    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 16:47:03.635481 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620252    4433 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 16:47:03.635481 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620256    4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 16:47:03.635481 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620259    4433 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 16:47:03.635481 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620262    4433 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 16:47:03.635481 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620265    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 16:47:03.635481 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620268    4433 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 16:47:03.635481 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620271    4433 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 16:47:03.635481 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620273    4433 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 16:47:03.635481 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620276    4433 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 16:47:03.635481 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620279    4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 16:47:03.635481 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620282    4433 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 16:47:03.635481 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620284    4433 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 16:47:03.635481 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620286    4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 16:47:03.635481 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620289    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 16:47:03.635481 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620292    4433 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 16:47:03.635481 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620294    4433 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 16:47:03.635952 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620296    4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 16:47:03.635952 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620299    4433 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 16:47:03.635952 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620301    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 16:47:03.635952 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620304    4433 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 16:47:03.635952 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620307    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 16:47:03.635952 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620309    4433 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 16:47:03.635952 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620312    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 16:47:03.635952 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620314    4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 16:47:03.635952 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620317    4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 16:47:03.635952 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620320    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 16:47:03.635952 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620322    4433 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 16:47:03.635952 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620325    4433 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 16:47:03.635952 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620328    4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 16:47:03.635952 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620331    4433 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 16:47:03.635952 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620333    4433 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 16:47:03.635952 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620336    4433 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 16:47:03.635952 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620338    4433 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 16:47:03.635952 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620341    4433 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 16:47:03.635952 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620344    4433 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 16:47:03.636423 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620347    4433 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 16:47:03.636423 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620349    4433 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 16:47:03.636423 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620352    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 16:47:03.636423 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620356    4433 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 16:47:03.636423 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620359    4433 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 16:47:03.636423 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620362    4433 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 16:47:03.636423 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620365    4433 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 16:47:03.636423 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620368    4433 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 16:47:03.636423 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620370    4433 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 16:47:03.636423 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620373    4433 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 16:47:03.636423 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620376    4433 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 16:47:03.636423 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620378    4433 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 16:47:03.636423 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620381    4433 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 16:47:03.636423 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620383    4433 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 16:47:03.636423 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620386    4433 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 16:47:03.636423 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620388    4433 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 16:47:03.636423 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620391    4433 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 16:47:03.636423 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620393    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 16:47:03.636423 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620396    4433 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 16:47:03.636895 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620399    4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 16:47:03.636895 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620415    4433 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 16:47:03.636895 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620418    4433 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 16:47:03.636895 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620421    4433 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 16:47:03.636895 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620423    4433 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 16:47:03.636895 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620426    4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 16:47:03.636895 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620429    4433 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 16:47:03.636895 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620432    4433 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 16:47:03.636895 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620434    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 16:47:03.636895 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620437    4433 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 16:47:03.636895 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620439    4433 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 16:47:03.636895 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620442    4433 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 16:47:03.636895 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620445    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 16:47:03.636895 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620449    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 16:47:03.636895 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620452    4433 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 16:47:03.636895 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.620475    4433 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 16:47:03.637283 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620507    4433 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 16:47:03.637283 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620511    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 16:47:03.637283 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620514    4433 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 16:47:03.637283 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620517    4433 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 16:47:03.637283 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620520    4433 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 16:47:03.637283 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620523    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 16:47:03.637283 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620525    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 16:47:03.637283 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620528    4433 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 16:47:03.637283 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620531    4433 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 16:47:03.637283 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620533    4433 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 16:47:03.637283 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620536    4433 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 16:47:03.637283 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620539    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 16:47:03.637283 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620541    4433 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 16:47:03.637283 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620544    4433 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 16:47:03.637283 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620547    4433 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 16:47:03.637283 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620549    4433 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 16:47:03.637283 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620552    4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 16:47:03.637283 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620555    4433 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 16:47:03.637283 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620557    4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 16:47:03.637283 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620560    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 16:47:03.637827 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620562    4433 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 16:47:03.637827 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620565    4433 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 16:47:03.637827 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620568    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 16:47:03.637827 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620570    4433 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 16:47:03.637827 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620573    4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 16:47:03.637827 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620576    4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 16:47:03.637827 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620578    4433 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 16:47:03.637827 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620581    4433 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 16:47:03.637827 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620584    4433 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 16:47:03.637827 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620587    4433 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 16:47:03.637827 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620590    4433 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 16:47:03.637827 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620593    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 16:47:03.637827 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620595    4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 16:47:03.637827 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620598    4433 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 16:47:03.637827 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620601    4433 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 16:47:03.637827 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620603    4433 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 16:47:03.637827 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620606    4433 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 16:47:03.637827 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620608    4433 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 16:47:03.637827 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620611    4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 16:47:03.638320 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620614    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 16:47:03.638320 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620616    4433 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 16:47:03.638320 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620619    4433 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 16:47:03.638320 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620621    4433 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 16:47:03.638320 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620625    4433 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 16:47:03.638320 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620629    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 16:47:03.638320 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620632    4433 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 16:47:03.638320 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620635    4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 16:47:03.638320 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620638    4433 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 16:47:03.638320 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620641    4433 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 16:47:03.638320 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620643    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 16:47:03.638320 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620646    4433 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 16:47:03.638320 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620649    4433 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 16:47:03.638320 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620652    4433 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 16:47:03.638320 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620654    4433 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 16:47:03.638320 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620657    4433 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 16:47:03.638320 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620660    4433 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 16:47:03.638320 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620662    4433 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 16:47:03.638320 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620665    4433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 16:47:03.638320 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620667    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 16:47:03.639000 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620670    4433 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 16:47:03.639000 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620675    4433 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 16:47:03.639000 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620677    4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 16:47:03.639000 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620680    4433 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 16:47:03.639000 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620683    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 16:47:03.639000 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620686    4433 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 16:47:03.639000 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620688    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 16:47:03.639000 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620691    4433 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 16:47:03.639000 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620693    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 16:47:03.639000 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620696    4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 16:47:03.639000 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620698    4433 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 16:47:03.639000 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620701    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 16:47:03.639000 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620703    4433 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 16:47:03.639000 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620706    4433 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 16:47:03.639000 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620709    4433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 16:47:03.639000 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620711    4433 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 16:47:03.639000 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620717    4433 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 16:47:03.639000 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620720    4433 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 16:47:03.639000 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620723    4433 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 16:47:03.639480 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620726    4433 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 16:47:03.639480 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620728    4433 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 16:47:03.639480 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620731    4433 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 16:47:03.639480 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620734    4433 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 16:47:03.639480 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620737    4433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 16:47:03.639480 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620741    4433 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 16:47:03.639480 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620744    4433 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 16:47:03.639480 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620748    4433 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 16:47:03.639480 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620751    4433 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 16:47:03.639480 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620754    4433 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 16:47:03.639480 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620756    4433 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 16:47:03.639480 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620759    4433 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 16:47:03.639480 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620762    4433 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 16:47:03.639480 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620764    4433 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 16:47:03.639480 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620768    4433 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 16:47:03.639480 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620770    4433 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 16:47:03.639480 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620773    4433 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 16:47:03.639480 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620776    4433 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 16:47:03.639480 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620785    4433 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 16:47:03.639480 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620788    4433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 16:47:03.640106 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620791    4433 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 16:47:03.640106 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620794    4433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 16:47:03.640106 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620796    4433 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 16:47:03.640106 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620799    4433 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 16:47:03.640106 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620802    4433 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 16:47:03.640106 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620804    4433 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 16:47:03.640106 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620807    4433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 16:47:03.640106 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:03.620809    4433 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 16:47:03.640106 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.620860    4433 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 16:47:03.640106 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.621865    4433 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 16:47:03.640106 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.621926    4433 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 16:47:03.640106 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.622148    4433 server.go:1014] "Starting client certificate rotation"
Oct 30 16:47:03.640106 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.622288    4433 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 16:47:03.640106 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.622359    4433 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 16:41:24 +0000 UTC" deadline="2028-07-23 03:15:44.862174868 +0000 UTC"
Oct 30 16:47:03.640106 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.622424    4433 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15154h28m41.239753685s"
Oct 30 16:47:03.640106 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.622711    4433 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 16:47:03.640541 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.623824    4433 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 16:47:03.640541 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.625189    4433 log.go:25] "Validated CRI v1 runtime API"
Oct 30 16:47:03.640541 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.626778    4433 log.go:25] "Validated CRI v1 image API"
Oct 30 16:47:03.640541 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.627533    4433 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 16:47:03.640541 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.629765    4433 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 dc6bcaed-293a-4f46-8c6d-e7787d9ac2a4:/dev/nvme0n1p3 de84304a-56b4-4fe4-82a0-36101d1a21a1:/dev/nvme0n1p4]
Oct 30 16:47:03.640791 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.629792    4433 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/01c9e0f62f817b50473b22a6c10db6d80f9c56e0ee401c5e1cd9d5422e5ae272/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/01c9e0f62f817b50473b22a6c10db6d80f9c56e0ee401c5e1cd9d5422e5ae272/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/37db11e32e134f2c89a5397054d650db55928e15f04054186705a8d79f6458a0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/37db11e32e134f2c89a5397054d650db55928e15f04054186705a8d79f6458a0/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b186c5c69966ccc90f463d36916ec22aa217e88fdd7539d03eac4722944334f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b186c5c69966ccc90f463d36916ec22aa217e88fdd7539d03eac4722944334f/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6fd6497c3149bc16e98b56ab74bcdf05d0cbe81c1705beee194a593125a0dbaa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6fd6497c3149bc16e98b56ab74bcdf05d0cbe81c1705beee194a593125a0dbaa/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80ab8702a92038db0be89a001a943928d4826d7222d85b3baad1fce938d6cf46/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80ab8702a92038db0be89a001a943928d4826d7222d85b3baad1fce938d6cf46/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/82bdbc4f6fde037dfb767e8ee70f82a0406ab4938445977074be2fef20f18ef7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/82bdbc4f6fde037dfb767e8ee70f82a0406ab4938445977074be2fef20f18ef7/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/89af626f67364a3694b3a603d4c9a6ef680340d7a0ae1c6776fa579c07bb61e1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/89af626f67364a3694b3a603d4c9a6ef680340d7a0ae1c6776fa579c07bb61e1/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/97014c2ed0aa27e2a90ced34214c7553866b01035fa740207826af713c3049ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/97014c2ed0aa27e2a90ced34214c7553866b01035fa740207826af713c3049ad/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a51865ae3b4dce0f9d9d528a37c4ec075132bcd9ef3d20d0a6dcdd87a786e3b6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a51865ae3b4dce0f9d9d528a37c4ec075132bcd9ef3d20d0a6dcdd87a786e3b6/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a987ba2e91dc2167bc9c214225be559a90d87b8b84f9ab76da87df360c9f4d3e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a987ba2e91dc2167bc9c214225be559a90d87b8b84f9ab76da87df360c9f4d3e/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b2e04792d09f6d7a2565deeb7ae4fdb20532e1b185bdfe090bb3185640d7d8b8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b2e04792d09f6d7a2565deeb7ae4fdb20532e1b185bdfe090bb3185640d7d8b8/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b7b4c2102de3367edb746a1e9cd1853a704af480c3fd6cb379ae7ab279125927/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b7b4c2102de3367edb746a1e9cd1853a704af480c3fd6cb379ae7ab279125927/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e82e8a05438606ab56ebf1b3cd180cb4f419dcabb8450e9fbb8165427ea18c1d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e82e8a05438606ab56ebf1b3cd180cb4f419dcabb8450e9fbb8165427ea18c1d/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ee74704cb44e6789528ec6ee598a4d54428d640f5cf90cf6496d0e8b95bbce93/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ee74704cb44e6789528ec6ee598a4d54428d640f5cf90cf6496d0e8b95bbce93/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03a9bc00-f973-4d50-ac32-a4ca843b977b/volumes/kubernetes.io~projected/kube-api-access-pxngf:{mountpoint:/var/lib/kubelet/pods/03a9bc00-f973-4d50-ac32-a4ca843b977b/volumes/kubernetes.io~projected/kube-api-access-pxngf major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3/volumes/kubernetes.io~projected/kube-api-access-vqjp6:{mountpoint:/var/lib/kubelet/pods/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3/volumes/kubernetes.io~projected/kube-api-access-vqjp6 major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33a051e9-adff-4247-8988-f6ea635aaee0/volumes/kubernetes.io~projected/kube-api-access-x97hn:{mountpoint:/var/lib/kubelet/pods/33a051e9-adff-4247-8988-f6ea635aaee0/volumes/kubernetes.io~projected/kube-api-access-x97hn major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47bd3f72-e568-4ded-ae3f-398eceff7033/volumes/kubernetes.io~projected/kube-api-access-j2fbm:{mountpoint:/var/lib/kubelet/pods/47bd3f72-e568-4ded-ae3f-398eceff7033/volumes/kubernetes.io~projected/kube-api-access-j2fbm major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47bd3f72-e568-4ded-ae3f-398eceff7033/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/47bd3f72-e568-4ded-ae3f-398eceff7033/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/498ef897-364c-4e19-9fbf-cdc46aac623f/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/498ef897-364c-4e19-9fbf-cdc46aac623f/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5cf6407b-d0de-4fa7-a346-1f211ab4c792/volumes/kubernetes.io~projected/kube-api-access-wgh6f:{mountpoint:/var/lib/kubelet/pods/5cf6407b-d0de-4fa7-a346-1f211ab4c792/volumes/kubernetes.io~projected/kube-api-access-wgh6f major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60154025-ad1c-41b0-b28c-1a83e5994685/volumes/kubernetes.io~projected/kube-api-access-f5l7w:{mountpoint:/var/lib/kubelet/pods/60154025-ad1c-41b0-b28c-1a83e5994685/volumes/kubernetes.io~projected/kube-api-access-f5l7w major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60f548f1-6eae-4929-bfd7-db1db9aab72e/volumes/kubernetes.io~projected/kube-api-access-k6jfp:{mountpoint:/var/lib/kubelet/pods/60f548f1-6eae-4929-bfd7-db1db9aab72e/volumes/kubernetes.io~projected/kube-api-access-k6jfp major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/631feafb-af15-4230-a5d7-86357d9d3953/volumes/kubernetes.io~projected/kube-api-access-cdnrp:{mountpoint:/var/lib/kubelet/pods/631feafb-af15-4230-a5d7-86357d9d3953/volumes/kubernetes.io~projected/kube-api-access-cdnrp major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c03c304-7756-45bb-9597-350026ed361e/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/6c03c304-7756-45bb-9597-350026ed361e/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7bfc195d-6fbc-45c8-9e2a-f9c206838651/volumes/kubernetes.io~projected/kube-api-access-jhxl7:{mountpoint:/var/lib/kubelet/pods/7bfc195d-6fbc-45c8-9e2a-f9c206838651/volumes/kubernetes.io~projected/kube-api-access-jhxl7 major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92b17f34-17e7-4d4f-b43a-4e828ede0187/volumes/kubernetes.io~projected/kube-api-access-r5wnk:{mountpoint:/var/lib/kubelet/pods/92b17f34-17e7-4d4f-b43a-4e828ede0187/volumes/kubernetes.io~projected/kube-api-access-r5wnk major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/980827e6-3d1b-4924-acde-8d408d5e50df/volumes/kubernetes.io~projected/kube-api-access-jch5v:{mountpoint:/var/lib/kubelet/pods/980827e6-3d1b-4924-acde-8d408d5e50df/volumes/kubernetes.io~projected/kube-api-access-jch5v major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf8a28a6-fd11-45dc-9927-adeba7780b46/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/cf8a28a6-fd11-45dc-9927-adeba7780b46/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf8a28a6-fd11-45dc-9927-adeba7780b46/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/cf8a28a6-fd11-45dc-9927-adeba7780b46/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf8a28a6-fd11-45dc-9927-adeba7780b46/volumes/kubernetes.io~projected/kube-api-access-vdd6c:{mountpoint:/var/lib/kubelet/pods/cf8a28a6-fd11-45dc-9927-adeba7780b46/volumes/kubernetes.io~projected/kube-api-access-vdd6c major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d90edf93-884d-46e3-b1f7-dcae29944ca2/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/d90edf93-884d-46e3-b1f7-dcae29944ca2/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d90edf93-884d-46e3-b1f7-dcae29944ca2/volumes/kubernetes.io~projected/kube-api-access-cxv5t:{mountpoint:/var/lib/kubelet/pods/d90edf93-884d-46e3-b1f7-dcae29944ca2/volumes/kubernetes.io~projected/kube-api-access-cxv5t major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d90edf93-884d-46e3-b1f7-dcae29944ca2/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/d90edf93-884d-46e3-b1f7-dcae29944ca2/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db037093-50f6-49b9-93c2-f37a7f1290d6/volumes/kubernetes.io~projected/kube-api-access-mkqgx:{mountpoint:/var/lib/kubelet/pods/db037093-50f6-49b9-93c2-f37a7f1290d6/volumes/kubernetes.io~projected/kube-api-access-mkqgx major:0 minor:59 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/f3372a83af96bcbe3fc3ff04ada288e44c7bf9588751cc8339a9ffd0e7cb6ba8/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/9d5f0dfa4ae007992ddbb7b392f2c97a41c423d4d72e26f0d56cfcf7fe203be9/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/320562b1cacf9390960d82cd2cde323ae6dd9f0a5d583d6e6d50d835c65259c4/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/956d734b87f3b9e5ffacaf2dc63f8ad5e1b7e966b31dcb32e549d9d72688a238/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/048f346a1409aef6a97f34a37592c03b9273d9a3191fbcbe9d58f646f3f24fe0/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/38f9313abd5e94c4a4c276b276c141e3fc42505b9fed6ab619e18a5514b20aeb/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/2201dd60dafee390e2c76cbd4f9f07fa6487d70b9b078fce88c19420e8c7b5d6/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/9bf92ae9b70d036eacf7f7aeb91f268a51f8ed99719c8211be2f0e605b8c2757/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/0669e27dc01c76d1cb60b28435a25fa4d59839fe959f92b7d9fdb738ad36e634/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/19bde4759a176a48a2cd608d82281905e4ff84d113f5e58e5eb0f0d0345e98b7/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/48cbe2bf143ea63cd057fe7dd5a98561a1fc9a7ab1fe9e0f092d3b289583f42b/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/a97f877e4e532f40044276b1a657deedcb269342127ea9aff8e97479bade1c71/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/a96f947be7f7d7a7cc2b3f59353d8d89025502e429365cf529f8bb6ac0353997/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/e979d5180ac831d7c4973258d424377a2740293203a9dd23df5e8f15c46a5f77/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/29013a370d8a27e0107af39c42308d7499517d98ed094d5723edf5c84970bde2/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/04bf32fe2a3a0c77c126ebb239d943fe2ad49bc7703e45e66afc9190e6121cab/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/8a7e5fc1d6917b4e0ed4e55f6339578067720549c4fddebacbc7e8a5230efcb7/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/8b3d0817f3c95d191d4730e4e6a2cd00fc0777d8899ac75209209ac72559b2ea/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/297f9f35b84af74c098373770343323010711ee750a29dede8372fe045811547/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/305ebd729efcdcb4aaaaf871a72898bc200b98ee891b7eea20174d69975110a1/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/7b5712e0cb88c981507a445cf871f0328dec2d0e45379de74657d09a41d13dc4/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/b915981213cb77fc77f9cfa8fc5e650893731b23c2c1839a52828e938396e497/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/d379894794aed1412181b8fd49b3d16fdfba947c35bd31633e795febe2e01490/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/0fe220e1035ee1468b3e24a76b26efeb5e440ef4f0cabf7166a9a1d234fc57ae/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/bfa34b6bce9292a3fc6dbfe4f0135ab77f37d1f0cd3a936ef76c6bd1d9e0d19c/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/ac9120193f87d5b8cb74454bde9e7a084652a4c9b298e2c831132857ea82e8d7/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/15762060205f240f20f37ace6dcf3feed134a010271b7a68c0f56293d95fa8e5/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/3ba57b477098865c640087cd96b89331c5a45e13c06ab49406bc9aa9cf686d84/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/cd7f0ebe85779bac424f29434099bdc22ef7029fb32e6fc2968768f4cc25f53b/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/93369da4df5130d4008d0bd3260fc6154864f2d57808dc7bea3a23de0c5c98ca/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/ff5ed137d69355fa0425a4b5e44b97c165a59374d97d700c0d734d14ce5cd060/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/5a313297224d5d5f6b28a7211b97a245be5f8764f75b344c3c56efa1ea65ad89/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/ba013b34e0ac5658ec86c9c2239b359c65f95ae521fae9b3e7589a3989dc6fe3/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/715b16a9452ebc415df98be9af13b6b372b9cff29dea216d935a011982ceeca9/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/f4b017520a70175b24242263eab167afed33a9ad7a353fd87c17555aee05ef92/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/c5693c7842f2a25f058bd9486a0ad4503b697c82c2d25e44bec40903685de335/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/c36b93f7e61e3d51d27dd92f3e56ae388deda0b7c8512c5cd8342810ac8aee69/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/a4abc10bfa1b4c498a9170c17250336fdfaa5831680f92a1ac8e28580a830058/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/e57b649165b28aeb77612ee19e0ceb87c125f474578a45849a25c3d121b9d373/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 16:47:03.640967 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.638360    4433 manager.go:217] Machine: {Timestamp:2026-10-30 16:47:03.637889871 +0000 UTC m=+0.072792070 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3091265 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ae7ffdff0fde93c8907da133c979c SystemUUID:ec2ae7ff-dff0-fde9-3c89-07da133c979c BootID:f25f2461-6b04-4036-94db-ce6afe849079 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6fd6497c3149bc16e98b56ab74bcdf05d0cbe81c1705beee194a593125a0dbaa/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/01c9e0f62f817b50473b22a6c10db6d80f9c56e0ee401c5e1cd9d5422e5ae272/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/47bd3f72-e568-4ded-ae3f-398eceff7033/volumes/kubernetes.io~projected/kube-api-access-j2fbm DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ee74704cb44e6789528ec6ee598a4d54428d640f5cf90cf6496d0e8b95bbce93/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/47bd3f72-e568-4ded-ae3f-398eceff7033/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d90edf93-884d-46e3-b1f7-dcae29944ca2/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/37db11e32e134f2c89a5397054d650db55928e15f04054186705a8d79f6458a0/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/89af626f67364a3694b3a603d4c9a6ef680340d7a0ae1c6776fa579c07bb61e1/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/33a051e9-adff-4247-8988-f6ea635aaee0/volumes/kubernetes.io~projected/kube-api-access-x97hn DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5cf6407b-d0de-4fa7-a346-1f211ab4c792/volumes/kubernetes.io~projected/kube-api-access-wgh6f DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/03a9bc00-f973-4d50-ac32-a4ca843b977b/volumes/kubernetes.io~projected/kube-api-access-pxngf DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7bfc195d-6fbc-45c8-9e2a-f9c206838651/volumes/kubernetes.io~projected/kube-api-access-jhxl7 DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e82e8a05438606ab56ebf1b3cd180cb4f419dcabb8450e9fbb8165427ea18c1d/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/92b17f34-17e7-4d4f-b43a-4e828ede0187/volumes/kubernetes.io~projected/kube-api-access-r5wnk DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/498ef897-364c-4e19-9fbf-cdc46aac623f/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/db037093-50f6-49b9-93c2-f37a7f1290d6/volumes/kubernetes.io~projected/kube-api-access-mkqgx DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80ab8702a92038db0be89a001a943928d4826d7222d85b3baad1fce938d6cf46/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3/volumes/kubernetes.io~projected/kube-api-access-vqjp6 DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cf8a28a6-fd11-45dc-9927-adeba7780b46/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cf8a28a6-fd11-45dc-9927-adeba7780b46/volumes/kubernetes.io~projected/kube-api-access-vdd6c DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d90edf93-884d-46e3-b1f7-dcae29944ca2/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/97014c2ed0aa27e2a90ced34214c7553866b01035fa740207826af713c3049ad/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a987ba2e91dc2167bc9c214225be559a90d87b8b84f9ab76da87df360c9f4d3e/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a51865ae3b4dce0f9d9d528a37c4ec075132bcd9ef3d20d0a6dcdd87a786e3b6/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/60f548f1-6eae-4929-bfd7-db1db9aab72e/volumes/kubernetes.io~projected/kube-api-access-k6jfp DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d90edf93-884d-46e3-b1f7-dcae29944ca2/volumes/kubernetes.io~projected/kube-api-access-cxv5t DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6c03c304-7756-45bb-9597-350026ed361e/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/631feafb-af15-4230-a5d7-86357d9d3953/volumes/kubernetes.io~projected/kube-api-access-cdnrp DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/82bdbc4f6fde037dfb767e8ee70f82a0406ab4938445977074be2fef20f18ef7/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b186c5c69966ccc90f463d36916ec22aa217e88fdd7539d03eac4722944334f/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/60154025-ad1c-41b0-b28c-1a83e5994685/volumes/kubernetes.io~projected/kube-api-access-f5l7w DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/980827e6-3d1b-4924-acde-8d408d5e50df/volumes/kubernetes.io~projected/kube-api-access-jch5v DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b7b4c2102de3367edb746a1e9cd1853a704af480c3fd6cb379ae7ab279125927/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b2e04792d09f6d7a2565deeb7ae4fdb20532e1b185bdfe090bb3185640d7d8b8/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cf8a28a6-fd11-45dc-9927-adeba7780b46/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:82bdbc4f6fde037 MacAddress:3a:5e:2b:0d:89:0b Speed:10000 Mtu:8901} {Name:97014c2ed0aa27e MacAddress:96:d2:e4:20:b2:57 Speed:10000 Mtu:8901} {Name:a51865ae3b4dce0 MacAddress:b2:a0:c8:10:00:07 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ea:b9:8d:51 Speed:0 Mtu:9001} {Name:br-int MacAddress:7e:ad:4f:2d:ad:40 Speed:0 Mtu:8901} {Name:e82e8a05438606a MacAddress:66:1e:da:f5:48:eb Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ea:b9:8d:51 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:1a:08:d9:3b:11:ca Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:d6:2a:94:49:57:e2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 16:47:03.640967 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.638611    4433 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 16:47:03.640967 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.638655    4433 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 16:47:03.640967 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.638990    4433 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 16:47:03.640967 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.639018    4433 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-13.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 16:47:03.640967 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.639150    4433 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 16:47:03.640967 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.639157    4433 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 16:47:03.640967 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.639167    4433 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 16:47:03.640967 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.639180    4433 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 16:47:03.640967 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.639249    4433 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 16:47:03.640967 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.639453    4433 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 16:47:03.640967 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.639558    4433 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 16:47:03.640967 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.639593    4433 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 16:47:03.640967 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.639603    4433 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 16:47:03.640967 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.639612    4433 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 16:47:03.640967 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.639621    4433 kubelet.go:405] "Adding apiserver pod source"
Oct 30 16:47:03.640967 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.639630    4433 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 16:47:03.640967 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.640346    4433 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 16:47:03.640967 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.640356    4433 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 16:47:03.640967 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.640897    4433 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 16:47:03.643388 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.643374    4433 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 16:47:03.643388 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.643387    4433 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 16:47:03.643536 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.643525    4433 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 16:47:03.643772 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.643760    4433 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 16:47:03.643805 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.643793    4433 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 16:47:03.643947 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.643931    4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 16:47:03.643986 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.643958    4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 16:47:03.643986 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.643972    4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 16:47:03.644036 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.643984    4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 16:47:03.644036 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.644005    4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 16:47:03.644036 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.644019    4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 16:47:03.644036 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.644031    4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 16:47:03.644159 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.644043    4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 16:47:03.644159 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.644057    4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 16:47:03.644159 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.644070    4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 16:47:03.644159 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.644097    4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 16:47:03.644159 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.644115    4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 16:47:03.644325 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.644162    4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 16:47:03.644325 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.644176    4433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 16:47:03.647687 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.647676    4433 server.go:1263] "Started kubelet"
Oct 30 16:47:03.647856 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.647812    4433 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 16:47:03.648207 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.647910    4433 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 16:47:03.648207 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.648018    4433 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 16:47:03.648220 ip-10-0-131-13 systemd[1]: Started Kubernetes Kubelet.
Oct 30 16:47:03.648472 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.648332    4433 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 16:47:03.653485 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.653466    4433 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 16:47:03.655051 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.655031    4433 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 16:47:03.655147 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.655104    4433 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 16:47:03.655183 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.655155    4433 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 16:41:26 +0000 UTC" deadline="2028-06-10 21:39:09.384386966 +0000 UTC"
Oct 30 16:47:03.655183 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.655173    4433 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14140h52m5.729216368s"
Oct 30 16:47:03.655312 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.655298    4433 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 16:47:03.655312 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.655311    4433 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 16:47:03.655432 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.655399    4433 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 16:47:03.655506 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:03.655488    4433 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-13.ec2.internal\" not found"
Oct 30 16:47:03.656741 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.656470    4433 factory.go:55] Registering systemd factory
Oct 30 16:47:03.656741 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.656495    4433 factory.go:223] Registration of the systemd container factory successfully
Oct 30 16:47:03.656935 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.656768    4433 factory.go:153] Registering CRI-O factory
Oct 30 16:47:03.656935 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.656780    4433 factory.go:223] Registration of the crio container factory successfully
Oct 30 16:47:03.656935 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.656801    4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03a9bc00-f973-4d50-ac32-a4ca843b977b" volumeName="kubernetes.io/configmap/03a9bc00-f973-4d50-ac32-a4ca843b977b-config-volume" seLinuxMountContext=""
Oct 30 16:47:03.656935 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.656842    4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c03c304-7756-45bb-9597-350026ed361e" volumeName="kubernetes.io/configmap/6c03c304-7756-45bb-9597-350026ed361e-konnectivity-ca" seLinuxMountContext=""
Oct 30 16:47:03.656935 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.656856    4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60154025-ad1c-41b0-b28c-1a83e5994685" volumeName="kubernetes.io/configmap/60154025-ad1c-41b0-b28c-1a83e5994685-multus-daemon-config" seLinuxMountContext=""
Oct 30 16:47:03.656935 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.656870    4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bfc195d-6fbc-45c8-9e2a-f9c206838651" volumeName="kubernetes.io/projected/7bfc195d-6fbc-45c8-9e2a-f9c206838651-kube-api-access-jhxl7" seLinuxMountContext=""
Oct 30 16:47:03.656935 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.656903    4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf8a28a6-fd11-45dc-9927-adeba7780b46" volumeName="kubernetes.io/empty-dir/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-tuned" seLinuxMountContext=""
Oct 30 16:47:03.656935 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.656912    4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d90edf93-884d-46e3-b1f7-dcae29944ca2" volumeName="kubernetes.io/configmap/d90edf93-884d-46e3-b1f7-dcae29944ca2-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 16:47:03.656935 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.656920    4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60154025-ad1c-41b0-b28c-1a83e5994685" volumeName="kubernetes.io/configmap/60154025-ad1c-41b0-b28c-1a83e5994685-cni-binary-copy" seLinuxMountContext=""
Oct 30 16:47:03.656935 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.656933    4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="631feafb-af15-4230-a5d7-86357d9d3953" volumeName="kubernetes.io/projected/631feafb-af15-4230-a5d7-86357d9d3953-kube-api-access-cdnrp" seLinuxMountContext=""
Oct 30 16:47:03.657433 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.656943    4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d90edf93-884d-46e3-b1f7-dcae29944ca2" volumeName="kubernetes.io/secret/d90edf93-884d-46e3-b1f7-dcae29944ca2-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 16:47:03.657433 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.656957    4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db037093-50f6-49b9-93c2-f37a7f1290d6" volumeName="kubernetes.io/projected/db037093-50f6-49b9-93c2-f37a7f1290d6-kube-api-access-mkqgx" seLinuxMountContext=""
Oct 30 16:47:03.657433 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.656966    4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03a9bc00-f973-4d50-ac32-a4ca843b977b" volumeName="kubernetes.io/projected/03a9bc00-f973-4d50-ac32-a4ca843b977b-kube-api-access-pxngf" seLinuxMountContext=""
Oct 30 16:47:03.657433 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.656973    4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b4a144d-1016-44d7-88ac-0e7c8db3e3c3" volumeName="kubernetes.io/empty-dir/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-data-volume" seLinuxMountContext=""
Oct 30 16:47:03.657433 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.656981    4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60f548f1-6eae-4929-bfd7-db1db9aab72e" volumeName="kubernetes.io/configmap/60f548f1-6eae-4929-bfd7-db1db9aab72e-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 16:47:03.657433 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.656989    4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c03c304-7756-45bb-9597-350026ed361e" volumeName="kubernetes.io/secret/6c03c304-7756-45bb-9597-350026ed361e-agent-certs" seLinuxMountContext=""
Oct 30 16:47:03.657433 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.656997    4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92b17f34-17e7-4d4f-b43a-4e828ede0187" volumeName="kubernetes.io/empty-dir/92b17f34-17e7-4d4f-b43a-4e828ede0187-tmp-dir" seLinuxMountContext=""
Oct 30 16:47:03.657433 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.657004    4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf8a28a6-fd11-45dc-9927-adeba7780b46" volumeName="kubernetes.io/empty-dir/cf8a28a6-fd11-45dc-9927-adeba7780b46-tmp" seLinuxMountContext=""
Oct 30 16:47:03.657433 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.657012    4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf8a28a6-fd11-45dc-9927-adeba7780b46" volumeName="kubernetes.io/projected/cf8a28a6-fd11-45dc-9927-adeba7780b46-kube-api-access-vdd6c" seLinuxMountContext=""
Oct 30 16:47:03.657433 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.657023    4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db037093-50f6-49b9-93c2-f37a7f1290d6" volumeName="kubernetes.io/configmap/db037093-50f6-49b9-93c2-f37a7f1290d6-serviceca" seLinuxMountContext=""
Oct 30 16:47:03.657433 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.657031    4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03a9bc00-f973-4d50-ac32-a4ca843b977b" volumeName="kubernetes.io/empty-dir/03a9bc00-f973-4d50-ac32-a4ca843b977b-tmp-dir" seLinuxMountContext=""
Oct 30 16:47:03.657433 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.657038    4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60f548f1-6eae-4929-bfd7-db1db9aab72e" volumeName="kubernetes.io/projected/60f548f1-6eae-4929-bfd7-db1db9aab72e-kube-api-access-k6jfp" seLinuxMountContext=""
Oct 30 16:47:03.657433 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.657045    4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92b17f34-17e7-4d4f-b43a-4e828ede0187" volumeName="kubernetes.io/projected/92b17f34-17e7-4d4f-b43a-4e828ede0187-kube-api-access-r5wnk" seLinuxMountContext=""
Oct 30 16:47:03.657433 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.657052    4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="980827e6-3d1b-4924-acde-8d408d5e50df" volumeName="kubernetes.io/projected/980827e6-3d1b-4924-acde-8d408d5e50df-kube-api-access-jch5v" seLinuxMountContext=""
Oct 30 16:47:03.657433 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.657060    4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47bd3f72-e568-4ded-ae3f-398eceff7033" volumeName="kubernetes.io/projected/47bd3f72-e568-4ded-ae3f-398eceff7033-kube-api-access-j2fbm" seLinuxMountContext=""
Oct 30 16:47:03.657433 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.657067    4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47bd3f72-e568-4ded-ae3f-398eceff7033" volumeName="kubernetes.io/secret/47bd3f72-e568-4ded-ae3f-398eceff7033-cert" seLinuxMountContext=""
Oct 30 16:47:03.657433 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.657075    4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="498ef897-364c-4e19-9fbf-cdc46aac623f" volumeName="kubernetes.io/secret/498ef897-364c-4e19-9fbf-cdc46aac623f-original-pull-secret" seLinuxMountContext=""
Oct 30 16:47:03.657433 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.657083    4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60154025-ad1c-41b0-b28c-1a83e5994685" volumeName="kubernetes.io/projected/60154025-ad1c-41b0-b28c-1a83e5994685-kube-api-access-f5l7w" seLinuxMountContext=""
Oct 30 16:47:03.657433 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.657090    4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60f548f1-6eae-4929-bfd7-db1db9aab72e" volumeName="kubernetes.io/configmap/60f548f1-6eae-4929-bfd7-db1db9aab72e-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 16:47:03.657433 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.657096    4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d90edf93-884d-46e3-b1f7-dcae29944ca2" volumeName="kubernetes.io/projected/d90edf93-884d-46e3-b1f7-dcae29944ca2-kube-api-access-cxv5t" seLinuxMountContext=""
Oct 30 16:47:03.657433 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.657108    4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33a051e9-adff-4247-8988-f6ea635aaee0" volumeName="kubernetes.io/configmap/33a051e9-adff-4247-8988-f6ea635aaee0-iptables-alerter-script" seLinuxMountContext=""
Oct 30 16:47:03.657433 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.657114    4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33a051e9-adff-4247-8988-f6ea635aaee0" volumeName="kubernetes.io/projected/33a051e9-adff-4247-8988-f6ea635aaee0-kube-api-access-x97hn" seLinuxMountContext=""
Oct 30 16:47:03.657433 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.657122    4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cf6407b-d0de-4fa7-a346-1f211ab4c792" volumeName="kubernetes.io/projected/5cf6407b-d0de-4fa7-a346-1f211ab4c792-kube-api-access-wgh6f" seLinuxMountContext=""
Oct 30 16:47:03.657433 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.657133    4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d90edf93-884d-46e3-b1f7-dcae29944ca2" volumeName="kubernetes.io/configmap/d90edf93-884d-46e3-b1f7-dcae29944ca2-env-overrides" seLinuxMountContext=""
Oct 30 16:47:03.657433 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.657140    4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b4a144d-1016-44d7-88ac-0e7c8db3e3c3" volumeName="kubernetes.io/projected/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-kube-api-access-vqjp6" seLinuxMountContext=""
Oct 30 16:47:03.657433 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.657148    4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60f548f1-6eae-4929-bfd7-db1db9aab72e" volumeName="kubernetes.io/configmap/60f548f1-6eae-4929-bfd7-db1db9aab72e-cni-binary-copy" seLinuxMountContext=""
Oct 30 16:47:03.657433 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.657156    4433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d90edf93-884d-46e3-b1f7-dcae29944ca2" volumeName="kubernetes.io/configmap/d90edf93-884d-46e3-b1f7-dcae29944ca2-ovnkube-config" seLinuxMountContext=""
Oct 30 16:47:03.657433 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.657166    4433 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 16:47:03.657433 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.657171    4433 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 16:47:03.657433 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.657438    4433 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 16:47:03.657433 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.657466    4433 factory.go:103] Registering Raw factory
Oct 30 16:47:03.658614 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.657482    4433 manager.go:1199] Started watching for new ooms in manager
Oct 30 16:47:03.658614 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.657849    4433 manager.go:319] Starting recovery of all containers
Oct 30 16:47:03.685995 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.685947    4433 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 16:47:03.687236 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.687211    4433 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 16:47:03.687352 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.687241    4433 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 16:47:03.687352 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.687269    4433 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 16:47:03.687352 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:03.687313    4433 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 16:47:03.697686 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.697655    4433 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 16:47:03.698808 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.698786    4433 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 16:47:03.701780 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.701756    4433 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 16:47:03.707582 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.707554    4433 generic.go:356] "Generic (PLEG): container finished" podID="91ca0575ad91df65b05568b935060571" containerID="bb1d71c86b64d98256f38c4eb7ef389b3ae6ccd113341dd92bfb6f29b9d0e0e8" exitCode=0
Oct 30 16:47:03.713260 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.713237    4433 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 16:47:03.713591 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.713567    4433 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 16:47:03.720571 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.720540    4433 generic.go:356] "Generic (PLEG): container finished" podID="60f548f1-6eae-4929-bfd7-db1db9aab72e" containerID="ac43b781f034b7d014799164cf0a166002f549398090db6b29bb2cbd9fd22508" exitCode=0
Oct 30 16:47:03.720571 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.720571    4433 generic.go:356] "Generic (PLEG): container finished" podID="60f548f1-6eae-4929-bfd7-db1db9aab72e" containerID="f9977fff0fcb6a43c42667ce17ee9dedb0e3e66194bc60a7d71209006a227dc7" exitCode=0
Oct 30 16:47:03.720719 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.720582    4433 generic.go:356] "Generic (PLEG): container finished" podID="60f548f1-6eae-4929-bfd7-db1db9aab72e" containerID="ec61f1997e8281023d67634750348f941347f0c9c0cdc19836184422061e4339" exitCode=0
Oct 30 16:47:03.720719 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.720590    4433 generic.go:356] "Generic (PLEG): container finished" podID="60f548f1-6eae-4929-bfd7-db1db9aab72e" containerID="e66caa8bf7c7b75c3c66a8c42b5c993bc89d740402b8707e0e638301df6b99db" exitCode=0
Oct 30 16:47:03.720719 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.720598    4433 generic.go:356] "Generic (PLEG): container finished" podID="60f548f1-6eae-4929-bfd7-db1db9aab72e" containerID="20a55b69d35871705839d6940e5be751878d235ff075d35aaa46d59330436ab7" exitCode=0
Oct 30 16:47:03.720719 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.720607    4433 generic.go:356] "Generic (PLEG): container finished" podID="60f548f1-6eae-4929-bfd7-db1db9aab72e" containerID="6cd0f0e8e2e1dcd5c2fcb9dbaca2dfa3b2aaf55c58a588dd17f7f191f2615274" exitCode=0
Oct 30 16:47:03.733180 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.733155    4433 manager.go:324] Recovery completed
Oct 30 16:47:03.747207 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.747180    4433 cpu_manager.go:226] "Starting" policy="none"
Oct 30 16:47:03.747207 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.747201    4433 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 16:47:03.747346 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.747224    4433 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 16:47:03.747446 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.747377    4433 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 16:47:03.747446 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.747393    4433 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 16:47:03.747522 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.747511    4433 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 16:47:03.747551 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.747522    4433 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 16:47:03.747551 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.747529    4433 policy_none.go:50] "Start"
Oct 30 16:47:03.747551 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.747539    4433 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 16:47:03.747629 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.747553    4433 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 16:47:03.760840 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.760783    4433 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 16:47:03.760840 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.760807    4433 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 16:47:03.760840 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.760815    4433 policy_none.go:44] "Start"
Oct 30 16:47:03.766046 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.766014    4433 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 16:47:03.769115 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.769099    4433 manager.go:341] "Starting Device Plugin manager"
Oct 30 16:47:03.769663 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:03.769127    4433 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 16:47:03.769663 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.769658    4433 server.go:87] "Starting device plugin registration server"
Oct 30 16:47:03.769834 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.769824    4433 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 16:47:03.769871 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.769834    4433 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 16:47:03.769996 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.769938    4433 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 16:47:03.770101 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.770048    4433 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 16:47:03.770101 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.770089    4433 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 16:47:03.770101 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.770096    4433 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 16:47:03.770306 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.770150    4433 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T16:47:03.770077772Z","UUID":"969b8a46-fd44-4d9a-b707-b7a42a4974e4","Handler":null,"Name":"","Endpoint":""}
Oct 30 16:47:03.771369 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:03.771347    4433 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 16:47:03.771715 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.771696    4433 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 16:47:03.771800 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.771723    4433 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 16:47:03.787421 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.787363    4433 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-13.ec2.internal"]
Oct 30 16:47:03.787837 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.787799    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-13.ec2.internal" event={"ID":"777e8481b0741b0fafcaf01b060f8274","Type":"ContainerStarted","Data":"aeea35f592c9272b07b8187a29883ea0b66cecac8282b0ab5466ff275d693aa0"}
Oct 30 16:47:03.787925 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.787847    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-13.ec2.internal" event={"ID":"777e8481b0741b0fafcaf01b060f8274","Type":"ContainerStarted","Data":"a987ba2e91dc2167bc9c214225be559a90d87b8b84f9ab76da87df360c9f4d3e"}
Oct 30 16:47:03.787925 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.787898    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal" event={"ID":"91ca0575ad91df65b05568b935060571","Type":"ContainerStarted","Data":"ef3aafe3b0b4756bfb0492d64c82b88fc04133349acd027a0570f348f86612b6"}
Oct 30 16:47:03.787925 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.787910    4433 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal"
Oct 30 16:47:03.788066 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.787911    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal" event={"ID":"91ca0575ad91df65b05568b935060571","Type":"ContainerDied","Data":"bb1d71c86b64d98256f38c4eb7ef389b3ae6ccd113341dd92bfb6f29b9d0e0e8"}
Oct 30 16:47:03.788066 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.788027    4433 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-13.ec2.internal"
Oct 30 16:47:03.788066 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.788028    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal" event={"ID":"91ca0575ad91df65b05568b935060571","Type":"ContainerStarted","Data":"6b186c5c69966ccc90f463d36916ec22aa217e88fdd7539d03eac4722944334f"}
Oct 30 16:47:03.809484 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.809459    4433 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 16:47:03.809572 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:03.809511    4433 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-13.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-13.ec2.internal"
Oct 30 16:47:03.809953 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.809941    4433 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 16:47:03.810012 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:03.809970    4433 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal"
Oct 30 16:47:03.870637 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.870608    4433 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 16:47:03.871872 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.871853    4433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-13.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 16:47:03.871983 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.871885    4433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-13.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 16:47:03.871983 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.871895    4433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-13.ec2.internal" event="NodeHasSufficientPID"
Oct 30 16:47:03.871983 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.871941    4433 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-13.ec2.internal"
Oct 30 16:47:03.898354 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.898327    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91ca0575ad91df65b05568b935060571-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal\" (UID: \"91ca0575ad91df65b05568b935060571\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal"
Oct 30 16:47:03.898479 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.898358    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/777e8481b0741b0fafcaf01b060f8274-config\") pod \"kube-apiserver-proxy-ip-10-0-131-13.ec2.internal\" (UID: \"777e8481b0741b0fafcaf01b060f8274\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-13.ec2.internal"
Oct 30 16:47:03.898479 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.898375    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91ca0575ad91df65b05568b935060571-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal\" (UID: \"91ca0575ad91df65b05568b935060571\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal"
Oct 30 16:47:03.905698 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.905677    4433 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-13.ec2.internal"
Oct 30 16:47:03.905860 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.905850    4433 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-13.ec2.internal"
Oct 30 16:47:03.999576 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.999539    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91ca0575ad91df65b05568b935060571-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal\" (UID: \"91ca0575ad91df65b05568b935060571\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal"
Oct 30 16:47:03.999576 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.999578    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/777e8481b0741b0fafcaf01b060f8274-config\") pod \"kube-apiserver-proxy-ip-10-0-131-13.ec2.internal\" (UID: \"777e8481b0741b0fafcaf01b060f8274\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-13.ec2.internal"
Oct 30 16:47:03.999771 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.999594    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91ca0575ad91df65b05568b935060571-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal\" (UID: \"91ca0575ad91df65b05568b935060571\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal"
Oct 30 16:47:03.999771 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.999668    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91ca0575ad91df65b05568b935060571-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal\" (UID: \"91ca0575ad91df65b05568b935060571\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal"
Oct 30 16:47:03.999771 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.999703    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91ca0575ad91df65b05568b935060571-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal\" (UID: \"91ca0575ad91df65b05568b935060571\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal"
Oct 30 16:47:03.999771 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:03.999668    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/777e8481b0741b0fafcaf01b060f8274-config\") pod \"kube-apiserver-proxy-ip-10-0-131-13.ec2.internal\" (UID: \"777e8481b0741b0fafcaf01b060f8274\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-13.ec2.internal"
Oct 30 16:47:04.640682 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.640647    4433 apiserver.go:52] "Watching apiserver"
Oct 30 16:47:04.723377 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.723344    4433 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-13.ec2.internal"
Oct 30 16:47:04.723377 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.723375    4433 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal"
Oct 30 16:47:04.733713 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.733688    4433 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 16:47:04.734094 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.734076    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-b46ww","openshift-network-diagnostics/network-check-target-r2pxh","openshift-ovn-kubernetes/ovnkube-node-kkb8g","kube-system/global-pull-secret-syncer-5cgn8","kube-system/kube-apiserver-proxy-ip-10-0-131-13.ec2.internal","openshift-multus/multus-additional-cni-plugins-bgwcr","openshift-dns/node-resolver-nlr68","openshift-network-operator/iptables-alerter-26t96","openshift-cluster-node-tuning-operator/tuned-f8wv9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal","kube-system/konnectivity-agent-vw5v7","openshift-dns/dns-default-5ztj2","openshift-insights/insights-runtime-extractor-mrllk","openshift-multus/multus-8t9lr","openshift-multus/network-metrics-daemon-7zf5f","openshift-image-registry/node-ca-jhqzq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf","openshift-ingress-canary/ingress-canary-ltvcq"]
Oct 30 16:47:04.738023 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.738004    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:47:04.738351 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.738338    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5ztj2"
Oct 30 16:47:04.741683 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.741663    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:47:04.743741 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.743722    4433 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 16:47:04.743826 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:04.743771    4433 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-13.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-13.ec2.internal"
Oct 30 16:47:04.748944 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.748926    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 16:47:04.749037 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.749007    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 16:47:04.749125 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.749037    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 16:47:04.749186 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.749173    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 16:47:04.749592 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.749579    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 16:47:04.749793 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.749779    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 16:47:04.749870 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.749856    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 16:47:04.749930 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.749895    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 16:47:04.750002 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.749991    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 16:47:04.750227 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.750210    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pdblj\""
Oct 30 16:47:04.750273 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.750230    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-8568q\""
Oct 30 16:47:04.750314 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.750301    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wd4nn\""
Oct 30 16:47:04.753903 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.753890    4433 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 16:47:04.753951 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:04.753927    4433 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-13.ec2.internal"
Oct 30 16:47:04.756861 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.756847    4433 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 16:47:04.767149 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.767122    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 16:47:04.767208 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.767159    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 16:47:04.767272 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.767213    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 16:47:04.767324 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.767303    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 16:47:04.767324 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.767312    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 16:47:04.767444 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.767358    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 16:47:04.767444 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.767373    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 16:47:04.767444 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.767382    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 16:47:04.767619 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.767516    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 16:47:04.767619 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.767569    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 16:47:04.767619 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.767575    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 16:47:04.767619 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.767576    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 16:47:04.767619 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.767570    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 16:47:04.767619 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.767614    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 16:47:04.767619 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.767633    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 16:47:04.767619 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.767636    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 16:47:04.767994 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.767676    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 16:47:04.767994 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.767725    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 16:47:04.767994 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.767751    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 16:47:04.767994 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.767760    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 16:47:04.767994 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.767769    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 16:47:04.767994 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.767808    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 16:47:04.767994 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.767807    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 16:47:04.767994 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.767845    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 16:47:04.767994 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.767847    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 16:47:04.767994 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.767864    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 16:47:04.804338 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.804307    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-run-k8s-cni-cncf-io\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.804338 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.804341    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:47:04.804491 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.804361    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03a9bc00-f973-4d50-ac32-a4ca843b977b-metrics-tls\") pod \"dns-default-5ztj2\" (UID: \"03a9bc00-f973-4d50-ac32-a4ca843b977b\") " pod="openshift-dns/dns-default-5ztj2"
Oct 30 16:47:04.804491 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.804392    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2fbm\" (UniqueName: \"kubernetes.io/projected/47bd3f72-e568-4ded-ae3f-398eceff7033-kube-api-access-j2fbm\") pod \"ingress-canary-ltvcq\" (UID: \"47bd3f72-e568-4ded-ae3f-398eceff7033\") " pod="openshift-ingress-canary/ingress-canary-ltvcq"
Oct 30 16:47:04.804491 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.804454    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxv5t\" (UniqueName: \"kubernetes.io/projected/d90edf93-884d-46e3-b1f7-dcae29944ca2-kube-api-access-cxv5t\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.804592 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.804570    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/60f548f1-6eae-4929-bfd7-db1db9aab72e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:47:04.804650 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.804633    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-sysconfig\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.804709 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.804690    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-tuned\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.804778 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.804765    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-tuned\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.804820 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.804780    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/498ef897-364c-4e19-9fbf-cdc46aac623f-kubelet-config\") pod \"global-pull-secret-syncer-5cgn8\" (UID: \"498ef897-364c-4e19-9fbf-cdc46aac623f\") " pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:47:04.804820 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.804813    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5wnk\" (UniqueName: \"kubernetes.io/projected/92b17f34-17e7-4d4f-b43a-4e828ede0187-kube-api-access-r5wnk\") pod \"node-resolver-nlr68\" (UID: \"92b17f34-17e7-4d4f-b43a-4e828ede0187\") " pod="openshift-dns/node-resolver-nlr68"
Oct 30 16:47:04.804886 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.804835    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-crio-socket\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:47:04.804886 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.804860    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/498ef897-364c-4e19-9fbf-cdc46aac623f-original-pull-secret\") pod \"global-pull-secret-syncer-5cgn8\" (UID: \"498ef897-364c-4e19-9fbf-cdc46aac623f\") " pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:47:04.804950 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.804904    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6c03c304-7756-45bb-9597-350026ed361e-agent-certs\") pod \"konnectivity-agent-vw5v7\" (UID: \"6c03c304-7756-45bb-9597-350026ed361e\") " pod="kube-system/konnectivity-agent-vw5v7"
Oct 30 16:47:04.804997 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.804976    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03a9bc00-f973-4d50-ac32-a4ca843b977b-tmp-dir\") pod \"dns-default-5ztj2\" (UID: \"03a9bc00-f973-4d50-ac32-a4ca843b977b\") " pod="openshift-dns/dns-default-5ztj2"
Oct 30 16:47:04.805029 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.805002    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/60f548f1-6eae-4929-bfd7-db1db9aab72e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:47:04.805029 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.805010    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-run-netns\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.805160 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.805038    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-run-ovn-kubernetes\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.805160 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.805062    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03a9bc00-f973-4d50-ac32-a4ca843b977b-tmp-dir\") pod \"dns-default-5ztj2\" (UID: \"03a9bc00-f973-4d50-ac32-a4ca843b977b\") " pod="openshift-dns/dns-default-5ztj2"
Oct 30 16:47:04.805160 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.805070    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:47:04.805160 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.805090    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-etc-selinux\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:47:04.805160 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.805133    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-var-lib-kubelet\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.805424 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.805170    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-hostroot\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.805424 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.805218    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-multus-socket-dir-parent\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.805424 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.805257    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-systemd-units\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.805424 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.805290    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-var-lib-openvswitch\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.805424 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.805318    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-etc-openvswitch\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.805424 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.805337    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d90edf93-884d-46e3-b1f7-dcae29944ca2-ovnkube-config\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.805424 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.805394    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkqgx\" (UniqueName: \"kubernetes.io/projected/db037093-50f6-49b9-93c2-f37a7f1290d6-kube-api-access-mkqgx\") pod \"node-ca-jhqzq\" (UID: \"db037093-50f6-49b9-93c2-f37a7f1290d6\") " pod="openshift-image-registry/node-ca-jhqzq"
Oct 30 16:47:04.805980 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.805446    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:47:04.805980 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.805476    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-systemd\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.805980 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.805532    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-run\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.805980 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.805556    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 16:47:04.805980 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.805560    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-run-openvswitch\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.805980 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.805584    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d90edf93-884d-46e3-b1f7-dcae29944ca2-env-overrides\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.805980 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.805593    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d90edf93-884d-46e3-b1f7-dcae29944ca2-ovnkube-config\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.805980 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.805613    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d90edf93-884d-46e3-b1f7-dcae29944ca2-ovnkube-script-lib\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.805980 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.805658    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-multus-cni-dir\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.805980 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.805661    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 16:47:04.805980 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.805700    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-var-lib-cni-multus\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.805980 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.805742    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47bd3f72-e568-4ded-ae3f-398eceff7033-cert\") pod \"ingress-canary-ltvcq\" (UID: \"47bd3f72-e568-4ded-ae3f-398eceff7033\") " pod="openshift-ingress-canary/ingress-canary-ltvcq"
Oct 30 16:47:04.805980 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.805826    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d90edf93-884d-46e3-b1f7-dcae29944ca2-ovnkube-script-lib\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.805980 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.805854    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60f548f1-6eae-4929-bfd7-db1db9aab72e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:47:04.805980 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.805868    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d90edf93-884d-46e3-b1f7-dcae29944ca2-env-overrides\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.805980 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.805883    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92b17f34-17e7-4d4f-b43a-4e828ede0187-tmp-dir\") pod \"node-resolver-nlr68\" (UID: \"92b17f34-17e7-4d4f-b43a-4e828ede0187\") " pod="openshift-dns/node-resolver-nlr68"
Oct 30 16:47:04.805980 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.805934    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-cnibin\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.805980 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806001    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8fhb8\""
Oct 30 16:47:04.805980 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806003    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-data-volume\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:47:04.805980 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806005    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92b17f34-17e7-4d4f-b43a-4e828ede0187-tmp-dir\") pod \"node-resolver-nlr68\" (UID: \"92b17f34-17e7-4d4f-b43a-4e828ede0187\") " pod="openshift-dns/node-resolver-nlr68"
Oct 30 16:47:04.805980 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806019    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9bdrw\""
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806008    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-vmkvz\""
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806050    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgh6f\" (UniqueName: \"kubernetes.io/projected/5cf6407b-d0de-4fa7-a346-1f211ab4c792-kube-api-access-wgh6f\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806077    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806109    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-data-volume\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806112    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-j4x52\""
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806154    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqjp6\" (UniqueName: \"kubernetes.io/projected/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-kube-api-access-vqjp6\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806157    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6bljg\""
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806183    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-run-netns\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806213    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/33a051e9-adff-4247-8988-f6ea635aaee0-iptables-alerter-script\") pod \"iptables-alerter-26t96\" (UID: \"33a051e9-adff-4247-8988-f6ea635aaee0\") " pod="openshift-network-operator/iptables-alerter-26t96"
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806239    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806245    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806243    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-z7xnd\""
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806249    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-device-dir\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806265    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rbwcw\""
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806278    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806302    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ffzh4\""
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806345    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806394    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60f548f1-6eae-4929-bfd7-db1db9aab72e-cnibin\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806451    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fxkbf\""
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806473    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60f548f1-6eae-4929-bfd7-db1db9aab72e-cni-binary-copy\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806492    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f9wpv\""
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806507    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhxl7\" (UniqueName: \"kubernetes.io/projected/7bfc195d-6fbc-45c8-9e2a-f9c206838651-kube-api-access-jhxl7\") pod \"network-check-target-r2pxh\" (UID: \"7bfc195d-6fbc-45c8-9e2a-f9c206838651\") " pod="openshift-network-diagnostics/network-check-target-r2pxh"
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806524    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/33a051e9-adff-4247-8988-f6ea635aaee0-iptables-alerter-script\") pod \"iptables-alerter-26t96\" (UID: \"33a051e9-adff-4247-8988-f6ea635aaee0\") " pod="openshift-network-operator/iptables-alerter-26t96"
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806543    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db037093-50f6-49b9-93c2-f37a7f1290d6-host\") pod \"node-ca-jhqzq\" (UID: \"db037093-50f6-49b9-93c2-f37a7f1290d6\") " pod="openshift-image-registry/node-ca-jhqzq"
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806572    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdnrp\" (UniqueName: \"kubernetes.io/projected/631feafb-af15-4230-a5d7-86357d9d3953-kube-api-access-cdnrp\") pod \"migrator-66df5d496b-b46ww\" (UID: \"631feafb-af15-4230-a5d7-86357d9d3953\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b46ww"
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806617    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7stlv\""
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806624    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-sysctl-conf\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806649    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f7c68\""
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806649    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60f548f1-6eae-4929-bfd7-db1db9aab72e-cni-binary-copy\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806667    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-cni-netd\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806675    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-szhrj\""
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806715    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/60154025-ad1c-41b0-b28c-1a83e5994685-multus-daemon-config\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806750    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/db037093-50f6-49b9-93c2-f37a7f1290d6-serviceca\") pod \"node-ca-jhqzq\" (UID: \"db037093-50f6-49b9-93c2-f37a7f1290d6\") " pod="openshift-image-registry/node-ca-jhqzq"
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806783    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-kubernetes\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806808    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-var-lib-kubelet\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806843    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-host\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806881    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33a051e9-adff-4247-8988-f6ea635aaee0-host-slash\") pod \"iptables-alerter-26t96\" (UID: \"33a051e9-adff-4247-8988-f6ea635aaee0\") " pod="openshift-network-operator/iptables-alerter-26t96"
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806903    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/03a9bc00-f973-4d50-ac32-a4ca843b977b-config-volume\") pod \"dns-default-5ztj2\" (UID: \"03a9bc00-f973-4d50-ac32-a4ca843b977b\") " pod="openshift-dns/dns-default-5ztj2"
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806940    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/60154025-ad1c-41b0-b28c-1a83e5994685-multus-daemon-config\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806941    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxngf\" (UniqueName: \"kubernetes.io/projected/03a9bc00-f973-4d50-ac32-a4ca843b977b-kube-api-access-pxngf\") pod \"dns-default-5ztj2\" (UID: \"03a9bc00-f973-4d50-ac32-a4ca843b977b\") " pod="openshift-dns/dns-default-5ztj2"
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806955    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/db037093-50f6-49b9-93c2-f37a7f1290d6-serviceca\") pod \"node-ca-jhqzq\" (UID: \"db037093-50f6-49b9-93c2-f37a7f1290d6\") " pod="openshift-image-registry/node-ca-jhqzq"
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806973    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60154025-ad1c-41b0-b28c-1a83e5994685-cni-binary-copy\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.806999    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x97hn\" (UniqueName: \"kubernetes.io/projected/33a051e9-adff-4247-8988-f6ea635aaee0-kube-api-access-x97hn\") pod \"iptables-alerter-26t96\" (UID: \"33a051e9-adff-4247-8988-f6ea635aaee0\") " pod="openshift-network-operator/iptables-alerter-26t96"
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.807023    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-run-systemd\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.807046    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-log-socket\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.807071    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/03a9bc00-f973-4d50-ac32-a4ca843b977b-config-volume\") pod \"dns-default-5ztj2\" (UID: \"03a9bc00-f973-4d50-ac32-a4ca843b977b\") " pod="openshift-dns/dns-default-5ztj2"
Oct 30 16:47:04.807132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.807071    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5l7w\" (UniqueName: \"kubernetes.io/projected/60154025-ad1c-41b0-b28c-1a83e5994685-kube-api-access-f5l7w\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.808848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.807206    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-var-lib-cni-bin\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.808848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.807210    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60154025-ad1c-41b0-b28c-1a83e5994685-cni-binary-copy\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.808848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.807246    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-sys\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.808848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.807271    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-slash\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.808848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.807288    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/60f548f1-6eae-4929-bfd7-db1db9aab72e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:47:04.808848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.807329    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/92b17f34-17e7-4d4f-b43a-4e828ede0187-hosts-file\") pod \"node-resolver-nlr68\" (UID: \"92b17f34-17e7-4d4f-b43a-4e828ede0187\") " pod="openshift-dns/node-resolver-nlr68"
Oct 30 16:47:04.808848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.807347    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-multus-conf-dir\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.808848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.807370    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs\") pod \"network-metrics-daemon-7zf5f\" (UID: \"980827e6-3d1b-4924-acde-8d408d5e50df\") " pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:47:04.808848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.807533    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6c03c304-7756-45bb-9597-350026ed361e-konnectivity-ca\") pod \"konnectivity-agent-vw5v7\" (UID: \"6c03c304-7756-45bb-9597-350026ed361e\") " pod="kube-system/konnectivity-agent-vw5v7"
Oct 30 16:47:04.808848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.807552    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-sys-fs\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:47:04.808848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.807570    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6jfp\" (UniqueName: \"kubernetes.io/projected/60f548f1-6eae-4929-bfd7-db1db9aab72e-kube-api-access-k6jfp\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:47:04.808848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.807587    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-etc-kubernetes\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.808848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.807642    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/60f548f1-6eae-4929-bfd7-db1db9aab72e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:47:04.808848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.807680    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdd6c\" (UniqueName: \"kubernetes.io/projected/cf8a28a6-fd11-45dc-9927-adeba7780b46-kube-api-access-vdd6c\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.808848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.807701    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-run-ovn\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.808848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.807721    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-cni-bin\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.808848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.807737    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d90edf93-884d-46e3-b1f7-dcae29944ca2-ovn-node-metrics-cert\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.808848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.807772    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60f548f1-6eae-4929-bfd7-db1db9aab72e-system-cni-dir\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:47:04.808848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.807871    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6c03c304-7756-45bb-9597-350026ed361e-konnectivity-ca\") pod \"konnectivity-agent-vw5v7\" (UID: \"6c03c304-7756-45bb-9597-350026ed361e\") " pod="kube-system/konnectivity-agent-vw5v7"
Oct 30 16:47:04.808848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.807886    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf8a28a6-fd11-45dc-9927-adeba7780b46-tmp\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.808848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.807910    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d90edf93-884d-46e3-b1f7-dcae29944ca2-ovn-node-metrics-cert\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.808848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.807941    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jch5v\" (UniqueName: \"kubernetes.io/projected/980827e6-3d1b-4924-acde-8d408d5e50df-kube-api-access-jch5v\") pod \"network-metrics-daemon-7zf5f\" (UID: \"980827e6-3d1b-4924-acde-8d408d5e50df\") " pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:47:04.808848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.807980    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/498ef897-364c-4e19-9fbf-cdc46aac623f-dbus\") pod \"global-pull-secret-syncer-5cgn8\" (UID: \"498ef897-364c-4e19-9fbf-cdc46aac623f\") " pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:47:04.808848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.807991    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf8a28a6-fd11-45dc-9927-adeba7780b46-tmp\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.808848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.808076    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.808848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.808131    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-socket-dir\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:47:04.808848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.808169    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-registration-dir\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:47:04.808848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.808201    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-system-cni-dir\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.808848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.808243    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-run-multus-certs\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.808848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.808273    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-modprobe-d\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.808848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.808288    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-sysctl-d\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.808848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.808303    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-lib-modules\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.808848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.808328    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-kubelet\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.808848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.808359    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-node-log\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.816164 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.816143    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47bd3f72-e568-4ded-ae3f-398eceff7033-cert\") pod \"ingress-canary-ltvcq\" (UID: \"47bd3f72-e568-4ded-ae3f-398eceff7033\") " pod="openshift-ingress-canary/ingress-canary-ltvcq"
Oct 30 16:47:04.816210 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.816133    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/498ef897-364c-4e19-9fbf-cdc46aac623f-original-pull-secret\") pod \"global-pull-secret-syncer-5cgn8\" (UID: \"498ef897-364c-4e19-9fbf-cdc46aac623f\") " pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:47:04.816210 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.816193    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6c03c304-7756-45bb-9597-350026ed361e-agent-certs\") pod \"konnectivity-agent-vw5v7\" (UID: \"6c03c304-7756-45bb-9597-350026ed361e\") " pod="kube-system/konnectivity-agent-vw5v7"
Oct 30 16:47:04.840980 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.840945    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdd6c\" (UniqueName: \"kubernetes.io/projected/cf8a28a6-fd11-45dc-9927-adeba7780b46-kube-api-access-vdd6c\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.841133 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.840975    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxngf\" (UniqueName: \"kubernetes.io/projected/03a9bc00-f973-4d50-ac32-a4ca843b977b-kube-api-access-pxngf\") pod \"dns-default-5ztj2\" (UID: \"03a9bc00-f973-4d50-ac32-a4ca843b977b\") " pod="openshift-dns/dns-default-5ztj2"
Oct 30 16:47:04.841133 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.840968    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x97hn\" (UniqueName: \"kubernetes.io/projected/33a051e9-adff-4247-8988-f6ea635aaee0-kube-api-access-x97hn\") pod \"iptables-alerter-26t96\" (UID: \"33a051e9-adff-4247-8988-f6ea635aaee0\") " pod="openshift-network-operator/iptables-alerter-26t96"
Oct 30 16:47:04.841133 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.840930    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jch5v\" (UniqueName: \"kubernetes.io/projected/980827e6-3d1b-4924-acde-8d408d5e50df-kube-api-access-jch5v\") pod \"network-metrics-daemon-7zf5f\" (UID: \"980827e6-3d1b-4924-acde-8d408d5e50df\") " pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:47:04.841366 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.841158    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2fbm\" (UniqueName: \"kubernetes.io/projected/47bd3f72-e568-4ded-ae3f-398eceff7033-kube-api-access-j2fbm\") pod \"ingress-canary-ltvcq\" (UID: \"47bd3f72-e568-4ded-ae3f-398eceff7033\") " pod="openshift-ingress-canary/ingress-canary-ltvcq"
Oct 30 16:47:04.841483 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.841442    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5wnk\" (UniqueName: \"kubernetes.io/projected/92b17f34-17e7-4d4f-b43a-4e828ede0187-kube-api-access-r5wnk\") pod \"node-resolver-nlr68\" (UID: \"92b17f34-17e7-4d4f-b43a-4e828ede0187\") " pod="openshift-dns/node-resolver-nlr68"
Oct 30 16:47:04.841483 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.840962    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxv5t\" (UniqueName: \"kubernetes.io/projected/d90edf93-884d-46e3-b1f7-dcae29944ca2-kube-api-access-cxv5t\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.841638 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.841613    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqjp6\" (UniqueName: \"kubernetes.io/projected/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-kube-api-access-vqjp6\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:47:04.841711 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.841680    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5l7w\" (UniqueName: \"kubernetes.io/projected/60154025-ad1c-41b0-b28c-1a83e5994685-kube-api-access-f5l7w\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.841761 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.841744    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdnrp\" (UniqueName: \"kubernetes.io/projected/631feafb-af15-4230-a5d7-86357d9d3953-kube-api-access-cdnrp\") pod \"migrator-66df5d496b-b46ww\" (UID: \"631feafb-af15-4230-a5d7-86357d9d3953\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-b46ww"
Oct 30 16:47:04.841851 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.841830    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6jfp\" (UniqueName: \"kubernetes.io/projected/60f548f1-6eae-4929-bfd7-db1db9aab72e-kube-api-access-k6jfp\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:47:04.841889 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.841855    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhxl7\" (UniqueName: \"kubernetes.io/projected/7bfc195d-6fbc-45c8-9e2a-f9c206838651-kube-api-access-jhxl7\") pod \"network-check-target-r2pxh\" (UID: \"7bfc195d-6fbc-45c8-9e2a-f9c206838651\") " pod="openshift-network-diagnostics/network-check-target-r2pxh"
Oct 30 16:47:04.845513 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.845488    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgh6f\" (UniqueName: \"kubernetes.io/projected/5cf6407b-d0de-4fa7-a346-1f211ab4c792-kube-api-access-wgh6f\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:47:04.845513 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.845502    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkqgx\" (UniqueName: \"kubernetes.io/projected/db037093-50f6-49b9-93c2-f37a7f1290d6-kube-api-access-mkqgx\") pod \"node-ca-jhqzq\" (UID: \"db037093-50f6-49b9-93c2-f37a7f1290d6\") " pod="openshift-image-registry/node-ca-jhqzq"
Oct 30 16:47:04.908718 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.908641    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-systemd-units\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.908718 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.908674    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-var-lib-openvswitch\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.908718 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.908690    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-etc-openvswitch\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.908718 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.908710    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:47:04.909021 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.908736    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-systemd\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.909021 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.908759    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-run\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.909021 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.908758    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-var-lib-openvswitch\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.909021 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.908770    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-etc-openvswitch\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.909021 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.908756    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-systemd-units\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.909021 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.908817    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-run\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.909021 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.908827    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-systemd\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.909021 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.908843    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-run-openvswitch\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.909021 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:04.908849    4433 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 16:47:04.909021 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.908876    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-run-openvswitch\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.909021 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.908881    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-multus-cni-dir\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.909021 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.908913    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-var-lib-cni-multus\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.909021 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:04.908947    4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-insights-runtime-extractor-tls podName:0b4a144d-1016-44d7-88ac-0e7c8db3e3c3 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:05.408924312 +0000 UTC m=+1.843826517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mrllk" (UID: "0b4a144d-1016-44d7-88ac-0e7c8db3e3c3") : secret "insights-runtime-extractor-tls" not found
Oct 30 16:47:04.909021 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.908960    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-var-lib-cni-multus\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.909021 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.908968    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60f548f1-6eae-4929-bfd7-db1db9aab72e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:47:04.909021 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.908998    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-cnibin\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.909021 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909020    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60f548f1-6eae-4929-bfd7-db1db9aab72e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:47:04.909021 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909024    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-run-netns\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909038    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-cnibin\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.908959    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-multus-cni-dir\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909053    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-device-dir\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909058    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-run-netns\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909079    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-device-dir\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909079    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60f548f1-6eae-4929-bfd7-db1db9aab72e-cnibin\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909102    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db037093-50f6-49b9-93c2-f37a7f1290d6-host\") pod \"node-ca-jhqzq\" (UID: \"db037093-50f6-49b9-93c2-f37a7f1290d6\") " pod="openshift-image-registry/node-ca-jhqzq"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909108    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60f548f1-6eae-4929-bfd7-db1db9aab72e-cnibin\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909117    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-sysctl-conf\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909133    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-cni-netd\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909148    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-kubernetes\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909152    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db037093-50f6-49b9-93c2-f37a7f1290d6-host\") pod \"node-ca-jhqzq\" (UID: \"db037093-50f6-49b9-93c2-f37a7f1290d6\") " pod="openshift-image-registry/node-ca-jhqzq"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909163    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-var-lib-kubelet\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909178    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-host\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909184    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-cni-netd\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909192    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33a051e9-adff-4247-8988-f6ea635aaee0-host-slash\") pod \"iptables-alerter-26t96\" (UID: \"33a051e9-adff-4247-8988-f6ea635aaee0\") " pod="openshift-network-operator/iptables-alerter-26t96"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909199    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-kubernetes\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909215    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33a051e9-adff-4247-8988-f6ea635aaee0-host-slash\") pod \"iptables-alerter-26t96\" (UID: \"33a051e9-adff-4247-8988-f6ea635aaee0\") " pod="openshift-network-operator/iptables-alerter-26t96"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909221    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-run-systemd\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909234    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-sysctl-conf\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909245    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-host\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909247    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-log-socket\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909252    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-var-lib-kubelet\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909266    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-run-systemd\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909279    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-log-socket\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909278    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-var-lib-cni-bin\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909305    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-sys\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909306    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-var-lib-cni-bin\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909319    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-slash\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909335    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/92b17f34-17e7-4d4f-b43a-4e828ede0187-hosts-file\") pod \"node-resolver-nlr68\" (UID: \"92b17f34-17e7-4d4f-b43a-4e828ede0187\") " pod="openshift-dns/node-resolver-nlr68"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909349    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-multus-conf-dir\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909360    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-sys\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909364    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs\") pod \"network-metrics-daemon-7zf5f\" (UID: \"980827e6-3d1b-4924-acde-8d408d5e50df\") " pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909374    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-slash\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909388    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-multus-conf-dir\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909419    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/92b17f34-17e7-4d4f-b43a-4e828ede0187-hosts-file\") pod \"node-resolver-nlr68\" (UID: \"92b17f34-17e7-4d4f-b43a-4e828ede0187\") " pod="openshift-dns/node-resolver-nlr68"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909428    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-sys-fs\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909447    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-etc-kubernetes\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:04.909450    4433 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909462    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-run-ovn\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909477    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-cni-bin\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909485    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-sys-fs\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:04.909504    4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs podName:980827e6-3d1b-4924-acde-8d408d5e50df nodeName:}" failed. No retries permitted until 2026-10-30 16:47:05.409489007 +0000 UTC m=+1.844391196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs") pod "network-metrics-daemon-7zf5f" (UID: "980827e6-3d1b-4924-acde-8d408d5e50df") : secret "metrics-daemon-secret" not found
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909503    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-cni-bin\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909529    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-etc-kubernetes\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909531    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-run-ovn\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909535    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60f548f1-6eae-4929-bfd7-db1db9aab72e-system-cni-dir\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909559    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60f548f1-6eae-4929-bfd7-db1db9aab72e-system-cni-dir\") pod \"multus-additional-cni-plugins-bgwcr\" (UID: \"60f548f1-6eae-4929-bfd7-db1db9aab72e\") " pod="openshift-multus/multus-additional-cni-plugins-bgwcr"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909562    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/498ef897-364c-4e19-9fbf-cdc46aac623f-dbus\") pod \"global-pull-secret-syncer-5cgn8\" (UID: \"498ef897-364c-4e19-9fbf-cdc46aac623f\") " pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909591    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909598    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/498ef897-364c-4e19-9fbf-cdc46aac623f-dbus\") pod \"global-pull-secret-syncer-5cgn8\" (UID: \"498ef897-364c-4e19-9fbf-cdc46aac623f\") " pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909610    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-socket-dir\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909615    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909627    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-registration-dir\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909644    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-system-cni-dir\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909658    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-run-multus-certs\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909672    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-modprobe-d\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909687    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-sysctl-d\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909656    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-socket-dir\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909702    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-lib-modules\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909715    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-registration-dir\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909734    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-run-multus-certs\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909754    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-system-cni-dir\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.909726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909784    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-kubelet\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.911657 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909807    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-node-log\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.911657 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909815    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-modprobe-d\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.911657 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909818    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-lib-modules\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.911657 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909831    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-run-k8s-cni-cncf-io\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.911657 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909827    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-sysctl-d\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.911657 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909841    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-kubelet\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.911657 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909850    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-node-log\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.911657 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909864    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:47:04.911657 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909873    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-run-k8s-cni-cncf-io\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.911657 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909889    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03a9bc00-f973-4d50-ac32-a4ca843b977b-metrics-tls\") pod \"dns-default-5ztj2\" (UID: \"03a9bc00-f973-4d50-ac32-a4ca843b977b\") " pod="openshift-dns/dns-default-5ztj2"
Oct 30 16:47:04.911657 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909908    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-sysconfig\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.911657 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:04.909927    4433 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 16:47:04.911657 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909968    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/498ef897-364c-4e19-9fbf-cdc46aac623f-kubelet-config\") pod \"global-pull-secret-syncer-5cgn8\" (UID: \"498ef897-364c-4e19-9fbf-cdc46aac623f\") " pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:47:04.911657 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909929    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/498ef897-364c-4e19-9fbf-cdc46aac623f-kubelet-config\") pod \"global-pull-secret-syncer-5cgn8\" (UID: \"498ef897-364c-4e19-9fbf-cdc46aac623f\") " pod="kube-system/global-pull-secret-syncer-5cgn8"
Oct 30 16:47:04.911657 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.909970    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cf8a28a6-fd11-45dc-9927-adeba7780b46-etc-sysconfig\") pod \"tuned-f8wv9\" (UID: \"cf8a28a6-fd11-45dc-9927-adeba7780b46\") " pod="openshift-cluster-node-tuning-operator/tuned-f8wv9"
Oct 30 16:47:04.911657 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:04.909992    4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-kube-rbac-proxy-cm podName:0b4a144d-1016-44d7-88ac-0e7c8db3e3c3 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:05.40997578 +0000 UTC m=+1.844877978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mrllk" (UID: "0b4a144d-1016-44d7-88ac-0e7c8db3e3c3") : configmap "kube-rbac-proxy" not found
Oct 30 16:47:04.911657 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:04.910002    4433 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 16:47:04.911657 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.910033    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-crio-socket\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:47:04.911657 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:04.910048    4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03a9bc00-f973-4d50-ac32-a4ca843b977b-metrics-tls podName:03a9bc00-f973-4d50-ac32-a4ca843b977b nodeName:}" failed. No retries permitted until 2026-10-30 16:47:05.410036723 +0000 UTC m=+1.844938922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/03a9bc00-f973-4d50-ac32-a4ca843b977b-metrics-tls") pod "dns-default-5ztj2" (UID: "03a9bc00-f973-4d50-ac32-a4ca843b977b") : secret "dns-default-metrics-tls" not found
Oct 30 16:47:04.911657 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.910077    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-crio-socket\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:47:04.911657 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.910077    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-run-netns\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.911657 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.910106    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-run-netns\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.911657 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.910115    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-run-ovn-kubernetes\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.911657 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.910139    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:47:04.911657 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.910162    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-etc-selinux\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:47:04.911657 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.910185    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-var-lib-kubelet\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.911657 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.910210    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-hostroot\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.911657 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.910236    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-multus-socket-dir-parent\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.911657 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.910242    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d90edf93-884d-46e3-b1f7-dcae29944ca2-host-run-ovn-kubernetes\") pod \"ovnkube-node-kkb8g\" (UID: \"d90edf93-884d-46e3-b1f7-dcae29944ca2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:04.911657 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.910285    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-etc-selinux\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:47:04.911657 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.910297    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-host-var-lib-kubelet\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.911657 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.910306    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5cf6407b-d0de-4fa7-a346-1f211ab4c792-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sx2qf\" (UID: \"5cf6407b-d0de-4fa7-a346-1f211ab4c792\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sx2qf"
Oct 30 16:47:04.911657 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.910315    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-hostroot\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:04.911657 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:04.910338    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60154025-ad1c-41b0-b28c-1a83e5994685-multus-socket-dir-parent\") pod \"multus-8t9lr\" (UID: \"60154025-ad1c-41b0-b28c-1a83e5994685\") " pod="openshift-multus/multus-8t9lr"
Oct 30 16:47:05.414290 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:05.414248    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:47:05.414290 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:05.414288    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03a9bc00-f973-4d50-ac32-a4ca843b977b-metrics-tls\") pod \"dns-default-5ztj2\" (UID: \"03a9bc00-f973-4d50-ac32-a4ca843b977b\") " pod="openshift-dns/dns-default-5ztj2"
Oct 30 16:47:05.414550 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:05.414322    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:47:05.414550 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:05.414369    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs\") pod \"network-metrics-daemon-7zf5f\" (UID: \"980827e6-3d1b-4924-acde-8d408d5e50df\") " pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:47:05.414550 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:05.414387    4433 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 16:47:05.414550 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:05.414442    4433 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 16:47:05.414550 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:05.414478    4433 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 16:47:05.414550 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:05.414483    4433 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 16:47:05.414550 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:05.414484    4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-kube-rbac-proxy-cm podName:0b4a144d-1016-44d7-88ac-0e7c8db3e3c3 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:06.414465189 +0000 UTC m=+2.849367390 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mrllk" (UID: "0b4a144d-1016-44d7-88ac-0e7c8db3e3c3") : configmap "kube-rbac-proxy" not found
Oct 30 16:47:05.414550 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:05.414533    4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03a9bc00-f973-4d50-ac32-a4ca843b977b-metrics-tls podName:03a9bc00-f973-4d50-ac32-a4ca843b977b nodeName:}" failed. No retries permitted until 2026-10-30 16:47:06.414522161 +0000 UTC m=+2.849424345 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/03a9bc00-f973-4d50-ac32-a4ca843b977b-metrics-tls") pod "dns-default-5ztj2" (UID: "03a9bc00-f973-4d50-ac32-a4ca843b977b") : secret "dns-default-metrics-tls" not found
Oct 30 16:47:05.414550 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:05.414542    4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs podName:980827e6-3d1b-4924-acde-8d408d5e50df nodeName:}" failed. No retries permitted until 2026-10-30 16:47:06.414536791 +0000 UTC m=+2.849438975 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs") pod "network-metrics-daemon-7zf5f" (UID: "980827e6-3d1b-4924-acde-8d408d5e50df") : secret "metrics-daemon-secret" not found
Oct 30 16:47:05.414550 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:05.414551    4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-insights-runtime-extractor-tls podName:0b4a144d-1016-44d7-88ac-0e7c8db3e3c3 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:06.414546399 +0000 UTC m=+2.849448583 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mrllk" (UID: "0b4a144d-1016-44d7-88ac-0e7c8db3e3c3") : secret "insights-runtime-extractor-tls" not found
Oct 30 16:47:05.842883 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:05.842856    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:06.422769 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:06.422736    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:47:06.422996 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:06.422785    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs\") pod \"network-metrics-daemon-7zf5f\" (UID: \"980827e6-3d1b-4924-acde-8d408d5e50df\") " pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:47:06.422996 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:06.422821    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:47:06.422996 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:06.422836    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03a9bc00-f973-4d50-ac32-a4ca843b977b-metrics-tls\") pod \"dns-default-5ztj2\" (UID: \"03a9bc00-f973-4d50-ac32-a4ca843b977b\") " pod="openshift-dns/dns-default-5ztj2"
Oct 30 16:47:06.422996 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:06.422899    4433 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 16:47:06.422996 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:06.422943    4433 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 16:47:06.422996 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:06.422962    4433 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 16:47:06.422996 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:06.422932    4433 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 16:47:06.422996 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:06.422992    4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-insights-runtime-extractor-tls podName:0b4a144d-1016-44d7-88ac-0e7c8db3e3c3 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:08.422972732 +0000 UTC m=+4.857874938 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mrllk" (UID: "0b4a144d-1016-44d7-88ac-0e7c8db3e3c3") : secret "insights-runtime-extractor-tls" not found
Oct 30 16:47:06.423278 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:06.423009    4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03a9bc00-f973-4d50-ac32-a4ca843b977b-metrics-tls podName:03a9bc00-f973-4d50-ac32-a4ca843b977b nodeName:}" failed. No retries permitted until 2026-10-30 16:47:08.422999355 +0000 UTC m=+4.857901559 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/03a9bc00-f973-4d50-ac32-a4ca843b977b-metrics-tls") pod "dns-default-5ztj2" (UID: "03a9bc00-f973-4d50-ac32-a4ca843b977b") : secret "dns-default-metrics-tls" not found
Oct 30 16:47:06.423278 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:06.423024    4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-kube-rbac-proxy-cm podName:0b4a144d-1016-44d7-88ac-0e7c8db3e3c3 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:08.423016628 +0000 UTC m=+4.857918812 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mrllk" (UID: "0b4a144d-1016-44d7-88ac-0e7c8db3e3c3") : configmap "kube-rbac-proxy" not found
Oct 30 16:47:06.423278 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:06.423038    4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs podName:980827e6-3d1b-4924-acde-8d408d5e50df nodeName:}" failed. No retries permitted until 2026-10-30 16:47:08.423029189 +0000 UTC m=+4.857931399 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs") pod "network-metrics-daemon-7zf5f" (UID: "980827e6-3d1b-4924-acde-8d408d5e50df") : secret "metrics-daemon-secret" not found
Oct 30 16:47:07.006323 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:07.006295    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nlr68_92b17f34-17e7-4d4f-b43a-4e828ede0187/dns-node-resolver/0.log"
Oct 30 16:47:08.007988 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:08.007958    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jhqzq_db037093-50f6-49b9-93c2-f37a7f1290d6/node-ca/0.log"
Oct 30 16:47:08.437351 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:08.437320    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs\") pod \"network-metrics-daemon-7zf5f\" (UID: \"980827e6-3d1b-4924-acde-8d408d5e50df\") " pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:47:08.437563 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:08.437369    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:47:08.437563 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:08.437390    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03a9bc00-f973-4d50-ac32-a4ca843b977b-metrics-tls\") pod \"dns-default-5ztj2\" (UID: \"03a9bc00-f973-4d50-ac32-a4ca843b977b\") " pod="openshift-dns/dns-default-5ztj2"
Oct 30 16:47:08.437563 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:08.437440    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:47:08.437563 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:08.437491    4433 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 16:47:08.437563 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:08.437549    4433 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 16:47:08.437805 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:08.437550    4433 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 16:47:08.437805 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:08.437562    4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-kube-rbac-proxy-cm podName:0b4a144d-1016-44d7-88ac-0e7c8db3e3c3 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:12.437546729 +0000 UTC m=+8.872448926 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mrllk" (UID: "0b4a144d-1016-44d7-88ac-0e7c8db3e3c3") : configmap "kube-rbac-proxy" not found
Oct 30 16:47:08.437805 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:08.437563    4433 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 16:47:08.437805 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:08.437674    4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03a9bc00-f973-4d50-ac32-a4ca843b977b-metrics-tls podName:03a9bc00-f973-4d50-ac32-a4ca843b977b nodeName:}" failed. No retries permitted until 2026-10-30 16:47:12.437655777 +0000 UTC m=+8.872557977 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/03a9bc00-f973-4d50-ac32-a4ca843b977b-metrics-tls") pod "dns-default-5ztj2" (UID: "03a9bc00-f973-4d50-ac32-a4ca843b977b") : secret "dns-default-metrics-tls" not found
Oct 30 16:47:08.437805 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:08.437693    4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-insights-runtime-extractor-tls podName:0b4a144d-1016-44d7-88ac-0e7c8db3e3c3 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:12.437682827 +0000 UTC m=+8.872585017 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mrllk" (UID: "0b4a144d-1016-44d7-88ac-0e7c8db3e3c3") : secret "insights-runtime-extractor-tls" not found
Oct 30 16:47:08.437805 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:08.437729    4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs podName:980827e6-3d1b-4924-acde-8d408d5e50df nodeName:}" failed. No retries permitted until 2026-10-30 16:47:12.437712622 +0000 UTC m=+8.872614821 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs") pod "network-metrics-daemon-7zf5f" (UID: "980827e6-3d1b-4924-acde-8d408d5e50df") : secret "metrics-daemon-secret" not found
Oct 30 16:47:08.607255 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:08.607212    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ltvcq_47bd3f72-e568-4ded-ae3f-398eceff7033/serve-healthcheck-canary/0.log"
Oct 30 16:47:09.008744 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:09.008708    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-b46ww_631feafb-af15-4230-a5d7-86357d9d3953/migrator/0.log"
Oct 30 16:47:12.463487 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:12.463450    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs\") pod \"network-metrics-daemon-7zf5f\" (UID: \"980827e6-3d1b-4924-acde-8d408d5e50df\") " pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:47:12.463908 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:12.463499    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:47:12.463908 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:12.463529    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03a9bc00-f973-4d50-ac32-a4ca843b977b-metrics-tls\") pod \"dns-default-5ztj2\" (UID: \"03a9bc00-f973-4d50-ac32-a4ca843b977b\") " pod="openshift-dns/dns-default-5ztj2"
Oct 30 16:47:12.463908 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:12.463572    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:47:12.463908 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:12.463673    4433 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 16:47:12.463908 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:12.463736    4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-insights-runtime-extractor-tls podName:0b4a144d-1016-44d7-88ac-0e7c8db3e3c3 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:20.463719145 +0000 UTC m=+16.898621329 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mrllk" (UID: "0b4a144d-1016-44d7-88ac-0e7c8db3e3c3") : secret "insights-runtime-extractor-tls" not found
Oct 30 16:47:12.463908 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:12.463672    4433 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 16:47:12.463908 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:12.463789    4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03a9bc00-f973-4d50-ac32-a4ca843b977b-metrics-tls podName:03a9bc00-f973-4d50-ac32-a4ca843b977b nodeName:}" failed. No retries permitted until 2026-10-30 16:47:20.463779129 +0000 UTC m=+16.898681315 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/03a9bc00-f973-4d50-ac32-a4ca843b977b-metrics-tls") pod "dns-default-5ztj2" (UID: "03a9bc00-f973-4d50-ac32-a4ca843b977b") : secret "dns-default-metrics-tls" not found
Oct 30 16:47:12.464167 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:12.463915    4433 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 16:47:12.464167 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:12.464077    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:47:12.465886 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:12.465868    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/980827e6-3d1b-4924-acde-8d408d5e50df-metrics-certs\") pod \"network-metrics-daemon-7zf5f\" (UID: \"980827e6-3d1b-4924-acde-8d408d5e50df\") " pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:47:12.539090 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:12.539054    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zf5f"
Oct 30 16:47:12.640137 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:12.640096    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-r2pxh"
Oct 30 16:47:12.832774 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:12.832693    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7zf5f"]
Oct 30 16:47:12.838236 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:12.838209    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod980827e6_3d1b_4924_acde_8d408d5e50df.slice/crio-dd343ed090265ccf0735aa9277e85a29413e8e9336bf1b70312664e54b9a971c WatchSource:0}: Error finding container dd343ed090265ccf0735aa9277e85a29413e8e9336bf1b70312664e54b9a971c: Status 404 returned error can't find the container with id dd343ed090265ccf0735aa9277e85a29413e8e9336bf1b70312664e54b9a971c
Oct 30 16:47:12.840387 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:12.840368    4433 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 16:47:13.720827 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:13.720802    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:13.721142 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:13.721048    4433 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 16:47:13.744543 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:13.744496    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7zf5f" event={"ID":"980827e6-3d1b-4924-acde-8d408d5e50df","Type":"ContainerStarted","Data":"dd343ed090265ccf0735aa9277e85a29413e8e9336bf1b70312664e54b9a971c"}
Oct 30 16:47:14.748921 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:14.748879    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7zf5f" event={"ID":"980827e6-3d1b-4924-acde-8d408d5e50df","Type":"ContainerStarted","Data":"1140ec28231276f3fb1b86684720f3c8d9270d0f0019d2d56c58e73829e842ad"}
Oct 30 16:47:14.749300 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:14.748974    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7zf5f" event={"ID":"980827e6-3d1b-4924-acde-8d408d5e50df","Type":"ContainerStarted","Data":"393f3d66a4a50b325b182c6647cf4cdab6d25f47f4d76a54a6a1bb6947212aa8"}
Oct 30 16:47:20.515760 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:20.515726    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-rtst9"]
Oct 30 16:47:20.518551 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:20.518523    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03a9bc00-f973-4d50-ac32-a4ca843b977b-metrics-tls\") pod \"dns-default-5ztj2\" (UID: \"03a9bc00-f973-4d50-ac32-a4ca843b977b\") " pod="openshift-dns/dns-default-5ztj2"
Oct 30 16:47:20.518636 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:20.518567    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:47:20.518779 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:20.518763    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cxb62"]
Oct 30 16:47:20.518904 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:20.518886    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-rtst9"
Oct 30 16:47:20.521062 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:20.521035    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b4a144d-1016-44d7-88ac-0e7c8db3e3c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mrllk\" (UID: \"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3\") " pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:47:20.521187 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:20.521088    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03a9bc00-f973-4d50-ac32-a4ca843b977b-metrics-tls\") pod \"dns-default-5ztj2\" (UID: \"03a9bc00-f973-4d50-ac32-a4ca843b977b\") " pod="openshift-dns/dns-default-5ztj2"
Oct 30 16:47:20.521535 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:20.521520    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cxb62"
Oct 30 16:47:20.544836 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:20.544813    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 16:47:20.545077 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:20.545060    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 16:47:20.556365 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:20.556334    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 16:47:20.556365 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:20.556350    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-6tvvp\""
Oct 30 16:47:20.556562 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:20.556384    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-8ps6v\""
Oct 30 16:47:20.556809 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:20.556794    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-rtst9"]
Oct 30 16:47:20.557316 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:20.557297    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cxb62"]
Oct 30 16:47:20.618933 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:20.618896    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd9cd8fd-063f-4f87-a80c-57f65f0d220b-tmp\") pod \"downloads-6dd789c8b7-rtst9\" (UID: \"dd9cd8fd-063f-4f87-a80c-57f65f0d220b\") " pod="openshift-console/downloads-6dd789c8b7-rtst9"
Oct 30 16:47:20.619115 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:20.618948    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-628kw\" (UniqueName: \"kubernetes.io/projected/dd9cd8fd-063f-4f87-a80c-57f65f0d220b-kube-api-access-628kw\") pod \"downloads-6dd789c8b7-rtst9\" (UID: \"dd9cd8fd-063f-4f87-a80c-57f65f0d220b\") " pod="openshift-console/downloads-6dd789c8b7-rtst9"
Oct 30 16:47:20.619115 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:20.619032    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ddfadf21-5be2-439c-af86-4c7ccf9545e7-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-cxb62\" (UID: \"ddfadf21-5be2-439c-af86-4c7ccf9545e7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cxb62"
Oct 30 16:47:20.639297 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:20.639259    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5ztj2"
Oct 30 16:47:20.643181 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:20.643158    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mrllk"
Oct 30 16:47:20.719886 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:20.719849    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd9cd8fd-063f-4f87-a80c-57f65f0d220b-tmp\") pod \"downloads-6dd789c8b7-rtst9\" (UID: \"dd9cd8fd-063f-4f87-a80c-57f65f0d220b\") " pod="openshift-console/downloads-6dd789c8b7-rtst9"
Oct 30 16:47:20.720042 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:20.719895    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-628kw\" (UniqueName: \"kubernetes.io/projected/dd9cd8fd-063f-4f87-a80c-57f65f0d220b-kube-api-access-628kw\") pod \"downloads-6dd789c8b7-rtst9\" (UID: \"dd9cd8fd-063f-4f87-a80c-57f65f0d220b\") " pod="openshift-console/downloads-6dd789c8b7-rtst9"
Oct 30 16:47:20.720042 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:20.719918    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ddfadf21-5be2-439c-af86-4c7ccf9545e7-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-cxb62\" (UID: \"ddfadf21-5be2-439c-af86-4c7ccf9545e7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cxb62"
Oct 30 16:47:20.721309 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:20.721276    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd9cd8fd-063f-4f87-a80c-57f65f0d220b-tmp\") pod \"downloads-6dd789c8b7-rtst9\" (UID: \"dd9cd8fd-063f-4f87-a80c-57f65f0d220b\") " pod="openshift-console/downloads-6dd789c8b7-rtst9"
Oct 30 16:47:20.723659 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:20.723638    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ddfadf21-5be2-439c-af86-4c7ccf9545e7-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-cxb62\" (UID: \"ddfadf21-5be2-439c-af86-4c7ccf9545e7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cxb62"
Oct 30 16:47:20.748910 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:20.748874    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-628kw\" (UniqueName: \"kubernetes.io/projected/dd9cd8fd-063f-4f87-a80c-57f65f0d220b-kube-api-access-628kw\") pod \"downloads-6dd789c8b7-rtst9\" (UID: \"dd9cd8fd-063f-4f87-a80c-57f65f0d220b\") " pod="openshift-console/downloads-6dd789c8b7-rtst9"
Oct 30 16:47:20.839531 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:20.839446    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-rtst9"
Oct 30 16:47:20.844547 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:20.844525    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cxb62"
Oct 30 16:47:20.954132 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:20.954102    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5ztj2"]
Oct 30 16:47:20.957848 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:20.957818    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a9bc00_f973_4d50_ac32_a4ca843b977b.slice/crio-f9bb2cf87e5e63cbdead7572928e68d8c0a308d21338b46595e1b429d1048ef1 WatchSource:0}: Error finding container f9bb2cf87e5e63cbdead7572928e68d8c0a308d21338b46595e1b429d1048ef1: Status 404 returned error can't find the container with id f9bb2cf87e5e63cbdead7572928e68d8c0a308d21338b46595e1b429d1048ef1
Oct 30 16:47:20.959447 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:20.959420    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mrllk"]
Oct 30 16:47:20.969699 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:20.969670    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b4a144d_1016_44d7_88ac_0e7c8db3e3c3.slice/crio-84eee46da830666227b3d1577d7a3ff55b5489edd3c63c4af504aa3746d7f7ef WatchSource:0}: Error finding container 84eee46da830666227b3d1577d7a3ff55b5489edd3c63c4af504aa3746d7f7ef: Status 404 returned error can't find the container with id 84eee46da830666227b3d1577d7a3ff55b5489edd3c63c4af504aa3746d7f7ef
Oct 30 16:47:21.144241 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:21.144159    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cxb62"]
Oct 30 16:47:21.148922 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:21.148891    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-rtst9"]
Oct 30 16:47:21.149546 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:21.149519    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddfadf21_5be2_439c_af86_4c7ccf9545e7.slice/crio-43bbf832ef493bba0edbbe0b4f4da7c6ff95bd478414b9cd5828c6cb68168960 WatchSource:0}: Error finding container 43bbf832ef493bba0edbbe0b4f4da7c6ff95bd478414b9cd5828c6cb68168960: Status 404 returned error can't find the container with id 43bbf832ef493bba0edbbe0b4f4da7c6ff95bd478414b9cd5828c6cb68168960
Oct 30 16:47:21.152661 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:21.152622    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd9cd8fd_063f_4f87_a80c_57f65f0d220b.slice/crio-6ba535f91aa6fcb703a19600da22d8e49e48b5786a8b8d29212791a328dca60e WatchSource:0}: Error finding container 6ba535f91aa6fcb703a19600da22d8e49e48b5786a8b8d29212791a328dca60e: Status 404 returned error can't find the container with id 6ba535f91aa6fcb703a19600da22d8e49e48b5786a8b8d29212791a328dca60e
Oct 30 16:47:21.770890 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:21.770843    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cxb62" event={"ID":"ddfadf21-5be2-439c-af86-4c7ccf9545e7","Type":"ContainerStarted","Data":"43bbf832ef493bba0edbbe0b4f4da7c6ff95bd478414b9cd5828c6cb68168960"}
Oct 30 16:47:21.772964 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:21.772913    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5ztj2" event={"ID":"03a9bc00-f973-4d50-ac32-a4ca843b977b","Type":"ContainerStarted","Data":"f9bb2cf87e5e63cbdead7572928e68d8c0a308d21338b46595e1b429d1048ef1"}
Oct 30 16:47:21.774766 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:21.774734    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-rtst9" event={"ID":"dd9cd8fd-063f-4f87-a80c-57f65f0d220b","Type":"ContainerStarted","Data":"6ba535f91aa6fcb703a19600da22d8e49e48b5786a8b8d29212791a328dca60e"}
Oct 30 16:47:21.777379 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:21.777266    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mrllk" event={"ID":"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3","Type":"ContainerStarted","Data":"eef9175ccc6f44286338f7151d2fb7bee7fb281637ed3d18d871a9e437e9dbbf"}
Oct 30 16:47:21.777379 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:21.777307    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mrllk" event={"ID":"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3","Type":"ContainerStarted","Data":"01ab7c7006a17c1527377667c433ff5d53f03c2fdd8bcdc08d3ba12254420b3a"}
Oct 30 16:47:21.777379 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:21.777329    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mrllk" event={"ID":"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3","Type":"ContainerStarted","Data":"84eee46da830666227b3d1577d7a3ff55b5489edd3c63c4af504aa3746d7f7ef"}
Oct 30 16:47:22.784435 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:22.783724    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cxb62" event={"ID":"ddfadf21-5be2-439c-af86-4c7ccf9545e7","Type":"ContainerStarted","Data":"3c58b19225dcd74db26361d772b9e3a55f3d5c5f733e041e105a290888982980"}
Oct 30 16:47:22.787048 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:22.785200    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cxb62"
Oct 30 16:47:22.791048 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:22.791025    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cxb62"
Oct 30 16:47:22.827987 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:22.827809    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cxb62" podStartSLOduration=1.754240526 podStartE2EDuration="2.827788664s" totalImagesPullingTime="1.073548138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:22.827249317 +0000 UTC m=+19.262151523" watchObservedRunningTime="2026-10-30 16:47:22.827788664 +0000 UTC m=+19.262690873"
Oct 30 16:47:23.431312 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:23.431277    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-xd2jw"]
Oct 30 16:47:23.434831 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:23.434801    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xd2jw"
Oct 30 16:47:23.452645 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:23.452620    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 16:47:23.452756 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:23.452619    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 16:47:23.452756 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:23.452683    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 16:47:23.452869 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:23.452821    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-zv954\""
Oct 30 16:47:23.452923 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:23.452867    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 16:47:23.452979 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:23.452940    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 16:47:23.464740 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:23.464716    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-xd2jw"]
Oct 30 16:47:23.539446 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:23.539398    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v4rs\" (UniqueName: \"kubernetes.io/projected/5a07cafb-21a7-4ad5-ab8d-34ee4d6d1394-kube-api-access-2v4rs\") pod \"prometheus-operator-9cd89b74c-xd2jw\" (UID: \"5a07cafb-21a7-4ad5-ab8d-34ee4d6d1394\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xd2jw"
Oct 30 16:47:23.539636 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:23.539512    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a07cafb-21a7-4ad5-ab8d-34ee4d6d1394-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-xd2jw\" (UID: \"5a07cafb-21a7-4ad5-ab8d-34ee4d6d1394\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xd2jw"
Oct 30 16:47:23.539636 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:23.539544    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a07cafb-21a7-4ad5-ab8d-34ee4d6d1394-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-xd2jw\" (UID: \"5a07cafb-21a7-4ad5-ab8d-34ee4d6d1394\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xd2jw"
Oct 30 16:47:23.539731 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:23.539631    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a07cafb-21a7-4ad5-ab8d-34ee4d6d1394-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-xd2jw\" (UID: \"5a07cafb-21a7-4ad5-ab8d-34ee4d6d1394\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xd2jw"
Oct 30 16:47:23.640351 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:23.640267    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a07cafb-21a7-4ad5-ab8d-34ee4d6d1394-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-xd2jw\" (UID: \"5a07cafb-21a7-4ad5-ab8d-34ee4d6d1394\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xd2jw"
Oct 30 16:47:23.640351 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:23.640322    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v4rs\" (UniqueName: \"kubernetes.io/projected/5a07cafb-21a7-4ad5-ab8d-34ee4d6d1394-kube-api-access-2v4rs\") pod \"prometheus-operator-9cd89b74c-xd2jw\" (UID: \"5a07cafb-21a7-4ad5-ab8d-34ee4d6d1394\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xd2jw"
Oct 30 16:47:23.640574 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:23.640374    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a07cafb-21a7-4ad5-ab8d-34ee4d6d1394-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-xd2jw\" (UID: \"5a07cafb-21a7-4ad5-ab8d-34ee4d6d1394\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xd2jw"
Oct 30 16:47:23.640574 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:23.640400    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a07cafb-21a7-4ad5-ab8d-34ee4d6d1394-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-xd2jw\" (UID: \"5a07cafb-21a7-4ad5-ab8d-34ee4d6d1394\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xd2jw"
Oct 30 16:47:23.641186 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:23.641156    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a07cafb-21a7-4ad5-ab8d-34ee4d6d1394-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-xd2jw\" (UID: \"5a07cafb-21a7-4ad5-ab8d-34ee4d6d1394\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xd2jw"
Oct 30 16:47:23.643398 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:23.643373    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a07cafb-21a7-4ad5-ab8d-34ee4d6d1394-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-xd2jw\" (UID: \"5a07cafb-21a7-4ad5-ab8d-34ee4d6d1394\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xd2jw"
Oct 30 16:47:23.643898 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:23.643879    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a07cafb-21a7-4ad5-ab8d-34ee4d6d1394-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-xd2jw\" (UID: \"5a07cafb-21a7-4ad5-ab8d-34ee4d6d1394\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xd2jw"
Oct 30 16:47:23.664637 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:23.664610    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v4rs\" (UniqueName: \"kubernetes.io/projected/5a07cafb-21a7-4ad5-ab8d-34ee4d6d1394-kube-api-access-2v4rs\") pod \"prometheus-operator-9cd89b74c-xd2jw\" (UID: \"5a07cafb-21a7-4ad5-ab8d-34ee4d6d1394\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xd2jw"
Oct 30 16:47:23.750433 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:23.750391    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xd2jw"
Oct 30 16:47:23.795585 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:23.795550    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mrllk" event={"ID":"0b4a144d-1016-44d7-88ac-0e7c8db3e3c3","Type":"ContainerStarted","Data":"49587b782f703ba541ac951b6ffdd3de8a38f0f9d792360b9b2049f8c201e02a"}
Oct 30 16:47:23.799093 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:23.798993    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5ztj2" event={"ID":"03a9bc00-f973-4d50-ac32-a4ca843b977b","Type":"ContainerStarted","Data":"597706e25cf598a6ac4a918c3e287bde4675e4f1bc6f2c1b48167d19bd2b319f"}
Oct 30 16:47:23.799093 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:23.799042    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5ztj2" event={"ID":"03a9bc00-f973-4d50-ac32-a4ca843b977b","Type":"ContainerStarted","Data":"394b1464bb5f324dce1bd102f568c14bd290f79e351e9948a5cf92fa27af3cd1"}
Oct 30 16:47:23.799305 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:23.799180    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5ztj2"
Oct 30 16:47:24.053801 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:24.053772    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-xd2jw"]
Oct 30 16:47:24.058333 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:24.058299    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a07cafb_21a7_4ad5_ab8d_34ee4d6d1394.slice/crio-403d0cbe5388456f15ce14e7be702ed0b8e5ec9cc9ebedbb6f663a046c191bc0 WatchSource:0}: Error finding container 403d0cbe5388456f15ce14e7be702ed0b8e5ec9cc9ebedbb6f663a046c191bc0: Status 404 returned error can't find the container with id 403d0cbe5388456f15ce14e7be702ed0b8e5ec9cc9ebedbb6f663a046c191bc0
Oct 30 16:47:24.813318 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:24.813275    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xd2jw" event={"ID":"5a07cafb-21a7-4ad5-ab8d-34ee4d6d1394","Type":"ContainerStarted","Data":"403d0cbe5388456f15ce14e7be702ed0b8e5ec9cc9ebedbb6f663a046c191bc0"}
Oct 30 16:47:25.298888 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:25.298848    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-597f9b54b6-nhbr9"]
Oct 30 16:47:25.302382 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:25.302357    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-597f9b54b6-nhbr9"
Oct 30 16:47:25.328518 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:25.328484    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hh9r6\""
Oct 30 16:47:25.328518 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:25.328498    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 16:47:25.328765 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:25.328572    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 16:47:25.328765 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:25.328584    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 16:47:25.328765 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:25.328614    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 16:47:25.328765 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:25.328706    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 16:47:25.329423 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:25.329337    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-597f9b54b6-nhbr9"]
Oct 30 16:47:25.453620 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:25.453583    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a9be69d-b840-4887-8a9a-822879938df1-oauth-serving-cert\") pod \"console-597f9b54b6-nhbr9\" (UID: \"5a9be69d-b840-4887-8a9a-822879938df1\") " pod="openshift-console/console-597f9b54b6-nhbr9"
Oct 30 16:47:25.453788 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:25.453635    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a9be69d-b840-4887-8a9a-822879938df1-console-config\") pod \"console-597f9b54b6-nhbr9\" (UID: \"5a9be69d-b840-4887-8a9a-822879938df1\") " pod="openshift-console/console-597f9b54b6-nhbr9"
Oct 30 16:47:25.453788 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:25.453662    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a9be69d-b840-4887-8a9a-822879938df1-console-serving-cert\") pod \"console-597f9b54b6-nhbr9\" (UID: \"5a9be69d-b840-4887-8a9a-822879938df1\") " pod="openshift-console/console-597f9b54b6-nhbr9"
Oct 30 16:47:25.453788 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:25.453734    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a9be69d-b840-4887-8a9a-822879938df1-tmp\") pod \"console-597f9b54b6-nhbr9\" (UID: \"5a9be69d-b840-4887-8a9a-822879938df1\") " pod="openshift-console/console-597f9b54b6-nhbr9"
Oct 30 16:47:25.453936 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:25.453855    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a9be69d-b840-4887-8a9a-822879938df1-console-oauth-config\") pod \"console-597f9b54b6-nhbr9\" (UID: \"5a9be69d-b840-4887-8a9a-822879938df1\") " pod="openshift-console/console-597f9b54b6-nhbr9"
Oct 30 16:47:25.453936 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:25.453884    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28h9w\" (UniqueName: \"kubernetes.io/projected/5a9be69d-b840-4887-8a9a-822879938df1-kube-api-access-28h9w\") pod \"console-597f9b54b6-nhbr9\" (UID: \"5a9be69d-b840-4887-8a9a-822879938df1\") " pod="openshift-console/console-597f9b54b6-nhbr9"
Oct 30 16:47:25.453936 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:25.453916    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a9be69d-b840-4887-8a9a-822879938df1-service-ca\") pod \"console-597f9b54b6-nhbr9\" (UID: \"5a9be69d-b840-4887-8a9a-822879938df1\") " pod="openshift-console/console-597f9b54b6-nhbr9"
Oct 30 16:47:25.554703 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:25.554674    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a9be69d-b840-4887-8a9a-822879938df1-console-oauth-config\") pod \"console-597f9b54b6-nhbr9\" (UID: \"5a9be69d-b840-4887-8a9a-822879938df1\") " pod="openshift-console/console-597f9b54b6-nhbr9"
Oct 30 16:47:25.554833 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:25.554713    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28h9w\" (UniqueName: \"kubernetes.io/projected/5a9be69d-b840-4887-8a9a-822879938df1-kube-api-access-28h9w\") pod \"console-597f9b54b6-nhbr9\" (UID: \"5a9be69d-b840-4887-8a9a-822879938df1\") " pod="openshift-console/console-597f9b54b6-nhbr9"
Oct 30 16:47:25.554833 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:25.554745    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a9be69d-b840-4887-8a9a-822879938df1-service-ca\") pod \"console-597f9b54b6-nhbr9\" (UID: \"5a9be69d-b840-4887-8a9a-822879938df1\") " pod="openshift-console/console-597f9b54b6-nhbr9"
Oct 30 16:47:25.554833 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:25.554782    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a9be69d-b840-4887-8a9a-822879938df1-oauth-serving-cert\") pod \"console-597f9b54b6-nhbr9\" (UID: \"5a9be69d-b840-4887-8a9a-822879938df1\") " pod="openshift-console/console-597f9b54b6-nhbr9"
Oct 30 16:47:25.555058 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:25.555036    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a9be69d-b840-4887-8a9a-822879938df1-console-config\") pod \"console-597f9b54b6-nhbr9\" (UID: \"5a9be69d-b840-4887-8a9a-822879938df1\") " pod="openshift-console/console-597f9b54b6-nhbr9"
Oct 30 16:47:25.555122 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:25.555078    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a9be69d-b840-4887-8a9a-822879938df1-console-serving-cert\") pod \"console-597f9b54b6-nhbr9\" (UID: \"5a9be69d-b840-4887-8a9a-822879938df1\") " pod="openshift-console/console-597f9b54b6-nhbr9"
Oct 30 16:47:25.555122 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:25.555093    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a9be69d-b840-4887-8a9a-822879938df1-tmp\") pod \"console-597f9b54b6-nhbr9\" (UID: \"5a9be69d-b840-4887-8a9a-822879938df1\") " pod="openshift-console/console-597f9b54b6-nhbr9"
Oct 30 16:47:25.555452 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:25.555430    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a9be69d-b840-4887-8a9a-822879938df1-tmp\") pod \"console-597f9b54b6-nhbr9\" (UID: \"5a9be69d-b840-4887-8a9a-822879938df1\") " pod="openshift-console/console-597f9b54b6-nhbr9"
Oct 30 16:47:25.555533 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:25.555432    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a9be69d-b840-4887-8a9a-822879938df1-oauth-serving-cert\") pod \"console-597f9b54b6-nhbr9\" (UID: \"5a9be69d-b840-4887-8a9a-822879938df1\") " pod="openshift-console/console-597f9b54b6-nhbr9"
Oct 30 16:47:25.555807 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:25.555779    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a9be69d-b840-4887-8a9a-822879938df1-console-config\") pod \"console-597f9b54b6-nhbr9\" (UID: \"5a9be69d-b840-4887-8a9a-822879938df1\") " pod="openshift-console/console-597f9b54b6-nhbr9"
Oct 30 16:47:25.555963 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:25.555940    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a9be69d-b840-4887-8a9a-822879938df1-service-ca\") pod \"console-597f9b54b6-nhbr9\" (UID: \"5a9be69d-b840-4887-8a9a-822879938df1\") " pod="openshift-console/console-597f9b54b6-nhbr9"
Oct 30 16:47:25.558185 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:25.558057    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a9be69d-b840-4887-8a9a-822879938df1-console-oauth-config\") pod \"console-597f9b54b6-nhbr9\" (UID: \"5a9be69d-b840-4887-8a9a-822879938df1\") " pod="openshift-console/console-597f9b54b6-nhbr9"
Oct 30 16:47:25.558344 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:25.558315    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a9be69d-b840-4887-8a9a-822879938df1-console-serving-cert\") pod \"console-597f9b54b6-nhbr9\" (UID: \"5a9be69d-b840-4887-8a9a-822879938df1\") " pod="openshift-console/console-597f9b54b6-nhbr9"
Oct 30 16:47:25.580463 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:25.580396    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28h9w\" (UniqueName: \"kubernetes.io/projected/5a9be69d-b840-4887-8a9a-822879938df1-kube-api-access-28h9w\") pod \"console-597f9b54b6-nhbr9\" (UID: \"5a9be69d-b840-4887-8a9a-822879938df1\") " pod="openshift-console/console-597f9b54b6-nhbr9"
Oct 30 16:47:25.617582 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:25.617547    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-597f9b54b6-nhbr9"
Oct 30 16:47:25.820563 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:25.820286    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xd2jw" event={"ID":"5a07cafb-21a7-4ad5-ab8d-34ee4d6d1394","Type":"ContainerStarted","Data":"596f036ca7a1752bc7075959cf0ced7aeb8382734c0e728ce7e5e2ef19818ff9"}
Oct 30 16:47:25.820563 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:25.820351    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xd2jw" event={"ID":"5a07cafb-21a7-4ad5-ab8d-34ee4d6d1394","Type":"ContainerStarted","Data":"29ee93bc1bcf9c117c4218e32d548cd2ffaae5de68d3050c29ceafacee778366"}
Oct 30 16:47:25.862357 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:25.862300    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xd2jw" podStartSLOduration=1.43614449 podStartE2EDuration="2.862281382s" totalImagesPullingTime="1.426136892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:25.861805018 +0000 UTC m=+22.296707225" watchObservedRunningTime="2026-10-30 16:47:25.862281382 +0000 UTC m=+22.297183592"
Oct 30 16:47:25.931130 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:25.931090    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-597f9b54b6-nhbr9"]
Oct 30 16:47:25.935882 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:25.935839    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a9be69d_b840_4887_8a9a_822879938df1.slice/crio-93194e6ef684e37f5b7fa4d9c9a9bfe71204b2e91ad7494a1eea4df5b6447c27 WatchSource:0}: Error finding container 93194e6ef684e37f5b7fa4d9c9a9bfe71204b2e91ad7494a1eea4df5b6447c27: Status 404 returned error can't find the container with id 93194e6ef684e37f5b7fa4d9c9a9bfe71204b2e91ad7494a1eea4df5b6447c27
Oct 30 16:47:26.827661 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:26.827619    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-597f9b54b6-nhbr9" event={"ID":"5a9be69d-b840-4887-8a9a-822879938df1","Type":"ContainerStarted","Data":"93194e6ef684e37f5b7fa4d9c9a9bfe71204b2e91ad7494a1eea4df5b6447c27"}
Oct 30 16:47:26.985910 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:26.985871    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 16:47:28.199353 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:28.199317    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cv7gw"]
Oct 30 16:47:28.206911 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:28.206459    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cv7gw"
Oct 30 16:47:28.227815 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:28.223429    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 16:47:28.232447 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:28.232420    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 16:47:28.232578 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:28.232429    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rb7tp\""
Oct 30 16:47:28.242528 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:28.242489    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 16:47:28.278787 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:28.278747    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7e04ca71-89c2-4232-9da2-7034affb6a99-root\") pod \"node-exporter-cv7gw\" (UID: \"7e04ca71-89c2-4232-9da2-7034affb6a99\") " pod="openshift-monitoring/node-exporter-cv7gw"
Oct 30 16:47:28.278959 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:28.278800    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7e04ca71-89c2-4232-9da2-7034affb6a99-node-exporter-wtmp\") pod \"node-exporter-cv7gw\" (UID: \"7e04ca71-89c2-4232-9da2-7034affb6a99\") " pod="openshift-monitoring/node-exporter-cv7gw"
Oct 30 16:47:28.278959 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:28.278844    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7e04ca71-89c2-4232-9da2-7034affb6a99-node-exporter-textfile\") pod \"node-exporter-cv7gw\" (UID: \"7e04ca71-89c2-4232-9da2-7034affb6a99\") " pod="openshift-monitoring/node-exporter-cv7gw"
Oct 30 16:47:28.278959 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:28.278872    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e04ca71-89c2-4232-9da2-7034affb6a99-metrics-client-ca\") pod \"node-exporter-cv7gw\" (UID: \"7e04ca71-89c2-4232-9da2-7034affb6a99\") " pod="openshift-monitoring/node-exporter-cv7gw"
Oct 30 16:47:28.278959 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:28.278918    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e04ca71-89c2-4232-9da2-7034affb6a99-sys\") pod \"node-exporter-cv7gw\" (UID: \"7e04ca71-89c2-4232-9da2-7034affb6a99\") " pod="openshift-monitoring/node-exporter-cv7gw"
Oct 30 16:47:28.278959 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:28.278955    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtzlg\" (UniqueName: \"kubernetes.io/projected/7e04ca71-89c2-4232-9da2-7034affb6a99-kube-api-access-vtzlg\") pod \"node-exporter-cv7gw\" (UID: \"7e04ca71-89c2-4232-9da2-7034affb6a99\") " pod="openshift-monitoring/node-exporter-cv7gw"
Oct 30 16:47:28.279195 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:28.278994    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7e04ca71-89c2-4232-9da2-7034affb6a99-node-exporter-tls\") pod \"node-exporter-cv7gw\" (UID: \"7e04ca71-89c2-4232-9da2-7034affb6a99\") " pod="openshift-monitoring/node-exporter-cv7gw"
Oct 30 16:47:28.279195 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:28.279022    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7e04ca71-89c2-4232-9da2-7034affb6a99-node-exporter-accelerators-collector-config\") pod \"node-exporter-cv7gw\" (UID: \"7e04ca71-89c2-4232-9da2-7034affb6a99\") " pod="openshift-monitoring/node-exporter-cv7gw"
Oct 30 16:47:28.279195 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:28.279052    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e04ca71-89c2-4232-9da2-7034affb6a99-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cv7gw\" (UID: \"7e04ca71-89c2-4232-9da2-7034affb6a99\") " pod="openshift-monitoring/node-exporter-cv7gw"
Oct 30 16:47:28.379814 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:28.379776    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7e04ca71-89c2-4232-9da2-7034affb6a99-node-exporter-textfile\") pod \"node-exporter-cv7gw\" (UID: \"7e04ca71-89c2-4232-9da2-7034affb6a99\") " pod="openshift-monitoring/node-exporter-cv7gw"
Oct 30 16:47:28.379814 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:28.379821    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e04ca71-89c2-4232-9da2-7034affb6a99-metrics-client-ca\") pod \"node-exporter-cv7gw\" (UID: \"7e04ca71-89c2-4232-9da2-7034affb6a99\") " pod="openshift-monitoring/node-exporter-cv7gw"
Oct 30 16:47:28.380063 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:28.379866    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e04ca71-89c2-4232-9da2-7034affb6a99-sys\") pod \"node-exporter-cv7gw\" (UID: \"7e04ca71-89c2-4232-9da2-7034affb6a99\") " pod="openshift-monitoring/node-exporter-cv7gw"
Oct 30 16:47:28.380063 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:28.379899    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtzlg\" (UniqueName: \"kubernetes.io/projected/7e04ca71-89c2-4232-9da2-7034affb6a99-kube-api-access-vtzlg\") pod \"node-exporter-cv7gw\" (UID: \"7e04ca71-89c2-4232-9da2-7034affb6a99\") " pod="openshift-monitoring/node-exporter-cv7gw"
Oct 30 16:47:28.380063 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:28.379934    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7e04ca71-89c2-4232-9da2-7034affb6a99-node-exporter-tls\") pod \"node-exporter-cv7gw\" (UID: \"7e04ca71-89c2-4232-9da2-7034affb6a99\") " pod="openshift-monitoring/node-exporter-cv7gw"
Oct 30 16:47:28.380063 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:28.379963    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7e04ca71-89c2-4232-9da2-7034affb6a99-node-exporter-accelerators-collector-config\") pod \"node-exporter-cv7gw\" (UID: \"7e04ca71-89c2-4232-9da2-7034affb6a99\") " pod="openshift-monitoring/node-exporter-cv7gw"
Oct 30 16:47:28.380063 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:28.379987    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e04ca71-89c2-4232-9da2-7034affb6a99-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cv7gw\" (UID: \"7e04ca71-89c2-4232-9da2-7034affb6a99\") " pod="openshift-monitoring/node-exporter-cv7gw"
Oct 30 16:47:28.380063 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:28.380008    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7e04ca71-89c2-4232-9da2-7034affb6a99-root\") pod \"node-exporter-cv7gw\" (UID: \"7e04ca71-89c2-4232-9da2-7034affb6a99\") " pod="openshift-monitoring/node-exporter-cv7gw"
Oct 30 16:47:28.380063 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:28.380021    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7e04ca71-89c2-4232-9da2-7034affb6a99-node-exporter-wtmp\") pod \"node-exporter-cv7gw\" (UID: \"7e04ca71-89c2-4232-9da2-7034affb6a99\") " pod="openshift-monitoring/node-exporter-cv7gw"
Oct 30 16:47:28.380296 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:28.380167    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7e04ca71-89c2-4232-9da2-7034affb6a99-node-exporter-wtmp\") pod \"node-exporter-cv7gw\" (UID: \"7e04ca71-89c2-4232-9da2-7034affb6a99\") " pod="openshift-monitoring/node-exporter-cv7gw"
Oct 30 16:47:28.380791 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:28.380436    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e04ca71-89c2-4232-9da2-7034affb6a99-sys\") pod \"node-exporter-cv7gw\" (UID: \"7e04ca71-89c2-4232-9da2-7034affb6a99\") " pod="openshift-monitoring/node-exporter-cv7gw"
Oct 30 16:47:28.380791 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:28.380475    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7e04ca71-89c2-4232-9da2-7034affb6a99-node-exporter-textfile\") pod \"node-exporter-cv7gw\" (UID: \"7e04ca71-89c2-4232-9da2-7034affb6a99\") " pod="openshift-monitoring/node-exporter-cv7gw"
Oct 30 16:47:28.380791 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:28.380758    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7e04ca71-89c2-4232-9da2-7034affb6a99-root\") pod \"node-exporter-cv7gw\" (UID: \"7e04ca71-89c2-4232-9da2-7034affb6a99\") " pod="openshift-monitoring/node-exporter-cv7gw"
Oct 30 16:47:28.381024 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:28.380991    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e04ca71-89c2-4232-9da2-7034affb6a99-metrics-client-ca\") pod \"node-exporter-cv7gw\" (UID: \"7e04ca71-89c2-4232-9da2-7034affb6a99\") " pod="openshift-monitoring/node-exporter-cv7gw"
Oct 30 16:47:28.381024 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:28.381003    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7e04ca71-89c2-4232-9da2-7034affb6a99-node-exporter-accelerators-collector-config\") pod \"node-exporter-cv7gw\" (UID: \"7e04ca71-89c2-4232-9da2-7034affb6a99\") " pod="openshift-monitoring/node-exporter-cv7gw"
Oct 30 16:47:28.383091 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:28.383055    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7e04ca71-89c2-4232-9da2-7034affb6a99-node-exporter-tls\") pod \"node-exporter-cv7gw\" (UID: \"7e04ca71-89c2-4232-9da2-7034affb6a99\") " pod="openshift-monitoring/node-exporter-cv7gw"
Oct 30 16:47:28.383540 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:28.383493    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e04ca71-89c2-4232-9da2-7034affb6a99-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cv7gw\" (UID: \"7e04ca71-89c2-4232-9da2-7034affb6a99\") " pod="openshift-monitoring/node-exporter-cv7gw"
Oct 30 16:47:28.404389 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:28.404358    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtzlg\" (UniqueName: \"kubernetes.io/projected/7e04ca71-89c2-4232-9da2-7034affb6a99-kube-api-access-vtzlg\") pod \"node-exporter-cv7gw\" (UID: \"7e04ca71-89c2-4232-9da2-7034affb6a99\") " pod="openshift-monitoring/node-exporter-cv7gw"
Oct 30 16:47:28.534301 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:28.534068    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cv7gw"
Oct 30 16:47:28.595481 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:28.595078    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e04ca71_89c2_4232_9da2_7034affb6a99.slice/crio-9f67291f7ed5e30438739e6d565cf511e092a09a7e6a2f4064528d1a9874f062 WatchSource:0}: Error finding container 9f67291f7ed5e30438739e6d565cf511e092a09a7e6a2f4064528d1a9874f062: Status 404 returned error can't find the container with id 9f67291f7ed5e30438739e6d565cf511e092a09a7e6a2f4064528d1a9874f062
Oct 30 16:47:28.835801 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:28.835725    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cv7gw" event={"ID":"7e04ca71-89c2-4232-9da2-7034affb6a99","Type":"ContainerStarted","Data":"9f67291f7ed5e30438739e6d565cf511e092a09a7e6a2f4064528d1a9874f062"}
Oct 30 16:47:29.646734 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.646699    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 16:47:29.650993 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.650969    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.667350 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.667325    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 16:47:29.667463 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.667365    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 16:47:29.667463 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.667445    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 16:47:29.667582 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.667484    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 16:47:29.667582 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.667501    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 16:47:29.667582 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.667528    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 16:47:29.667948 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.667611    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-29g2c\""
Oct 30 16:47:29.691082 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.691020    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 16:47:29.691082 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.691022    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 16:47:29.691285 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.691099    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 16:47:29.693931 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.693910    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 16:47:29.791729 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.791642    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-web-config\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.791891 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.791780    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66f44d22-9ae0-44aa-a585-cf1370bdb459-tls-assets\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.791891 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.791827    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.791891 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.791863    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66f44d22-9ae0-44aa-a585-cf1370bdb459-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.792055 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.791962    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.792055 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.791998    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66f44d22-9ae0-44aa-a585-cf1370bdb459-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.792135 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.792074    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-config-volume\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.792135 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.792115    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45rz9\" (UniqueName: \"kubernetes.io/projected/66f44d22-9ae0-44aa-a585-cf1370bdb459-kube-api-access-45rz9\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.792215 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.792154    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.792258 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.792185    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.792258 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.792243    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66f44d22-9ae0-44aa-a585-cf1370bdb459-config-out\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.792355 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.792270    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.792355 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.792321    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/66f44d22-9ae0-44aa-a585-cf1370bdb459-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.841619 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.841576    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-597f9b54b6-nhbr9" event={"ID":"5a9be69d-b840-4887-8a9a-822879938df1","Type":"ContainerStarted","Data":"6df394d9d4f038b4014ba4af6934d6e06f4cd283f17da6b8edbd1a07218ad8f3"}
Oct 30 16:47:29.843452 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.843392    4433 generic.go:356] "Generic (PLEG): container finished" podID="7e04ca71-89c2-4232-9da2-7034affb6a99" containerID="f0ba555eb66937f9d5de91ae12271cf6723dbfba83249d943e01fb36a684b17c" exitCode=0
Oct 30 16:47:29.843452 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.843438    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cv7gw" event={"ID":"7e04ca71-89c2-4232-9da2-7034affb6a99","Type":"ContainerDied","Data":"f0ba555eb66937f9d5de91ae12271cf6723dbfba83249d943e01fb36a684b17c"}
Oct 30 16:47:29.885586 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.885523    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-597f9b54b6-nhbr9" podStartSLOduration=1.978111071 podStartE2EDuration="4.885505661s" totalImagesPullingTime="2.90739459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:29.884365508 +0000 UTC m=+26.319267713" watchObservedRunningTime="2026-10-30 16:47:29.885505661 +0000 UTC m=+26.320407867"
Oct 30 16:47:29.893316 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.893285    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45rz9\" (UniqueName: \"kubernetes.io/projected/66f44d22-9ae0-44aa-a585-cf1370bdb459-kube-api-access-45rz9\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.893478 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.893339    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.893478 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.893383    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.893478 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.893426    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66f44d22-9ae0-44aa-a585-cf1370bdb459-config-out\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.893478 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.893454    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.893633 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.893479    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/66f44d22-9ae0-44aa-a585-cf1370bdb459-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.893633 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.893535    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-web-config\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.893633 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.893583    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66f44d22-9ae0-44aa-a585-cf1370bdb459-tls-assets\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.893633 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.893613    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.893780 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.893652    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66f44d22-9ae0-44aa-a585-cf1370bdb459-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.893780 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.893719    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.893780 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.893764    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66f44d22-9ae0-44aa-a585-cf1370bdb459-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.893870 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.893792    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-config-volume\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.894015 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.893991    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/66f44d22-9ae0-44aa-a585-cf1370bdb459-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.897005 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.895519    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66f44d22-9ae0-44aa-a585-cf1370bdb459-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.897005 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.896693    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.897005 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.896773    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.897005 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.896959    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66f44d22-9ae0-44aa-a585-cf1370bdb459-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.897005 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.896971    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-config-volume\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.897773 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.897743    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.897880 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.897794    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-web-config\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.897880 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.897813    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.898002 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.897910    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66f44d22-9ae0-44aa-a585-cf1370bdb459-tls-assets\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.898002 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.897995    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66f44d22-9ae0-44aa-a585-cf1370bdb459-config-out\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.899655 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.899631    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.917387 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.917357    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45rz9\" (UniqueName: \"kubernetes.io/projected/66f44d22-9ae0-44aa-a585-cf1370bdb459-kube-api-access-45rz9\") pod \"alertmanager-main-0\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:29.979490 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:29.979459    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:47:30.321758 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.321683    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 16:47:30.325577 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:30.325551    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66f44d22_9ae0_44aa_a585_cf1370bdb459.slice/crio-dbd4c657a6c968507fafd6096d62c9ad911bc72c8ba5919992700c5ac5934e46 WatchSource:0}: Error finding container dbd4c657a6c968507fafd6096d62c9ad911bc72c8ba5919992700c5ac5934e46: Status 404 returned error can't find the container with id dbd4c657a6c968507fafd6096d62c9ad911bc72c8ba5919992700c5ac5934e46
Oct 30 16:47:30.402236 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.402195    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7c8f575fb-jsx25"]
Oct 30 16:47:30.407891 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.407870    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25"
Oct 30 16:47:30.427031 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.426983    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 16:47:30.427314 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.427295    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-wxq7r\""
Oct 30 16:47:30.428441 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.427745    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 16:47:30.428441 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.428049    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 16:47:30.428441 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.428305    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-1iq3ks2f47ani\""
Oct 30 16:47:30.428663 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.428514    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 16:47:30.428913 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.428893    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 16:47:30.434268 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.434246    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7c8f575fb-jsx25"]
Oct 30 16:47:30.500298 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.500258    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/52a9edcf-c077-4650-844b-24a3c6a0f87e-secret-grpc-tls\") pod \"thanos-querier-7c8f575fb-jsx25\" (UID: \"52a9edcf-c077-4650-844b-24a3c6a0f87e\") " pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25"
Oct 30 16:47:30.500498 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.500321    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/52a9edcf-c077-4650-844b-24a3c6a0f87e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c8f575fb-jsx25\" (UID: \"52a9edcf-c077-4650-844b-24a3c6a0f87e\") " pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25"
Oct 30 16:47:30.500498 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.500424    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnktd\" (UniqueName: \"kubernetes.io/projected/52a9edcf-c077-4650-844b-24a3c6a0f87e-kube-api-access-tnktd\") pod \"thanos-querier-7c8f575fb-jsx25\" (UID: \"52a9edcf-c077-4650-844b-24a3c6a0f87e\") " pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25"
Oct 30 16:47:30.500498 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.500470    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52a9edcf-c077-4650-844b-24a3c6a0f87e-tmp\") pod \"thanos-querier-7c8f575fb-jsx25\" (UID: \"52a9edcf-c077-4650-844b-24a3c6a0f87e\") " pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25"
Oct 30 16:47:30.500674 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.500497    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/52a9edcf-c077-4650-844b-24a3c6a0f87e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c8f575fb-jsx25\" (UID: \"52a9edcf-c077-4650-844b-24a3c6a0f87e\") " pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25"
Oct 30 16:47:30.500674 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.500559    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52a9edcf-c077-4650-844b-24a3c6a0f87e-metrics-client-ca\") pod \"thanos-querier-7c8f575fb-jsx25\" (UID: \"52a9edcf-c077-4650-844b-24a3c6a0f87e\") " pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25"
Oct 30 16:47:30.500774 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.500705    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/52a9edcf-c077-4650-844b-24a3c6a0f87e-secret-thanos-querier-tls\") pod \"thanos-querier-7c8f575fb-jsx25\" (UID: \"52a9edcf-c077-4650-844b-24a3c6a0f87e\") " pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25"
Oct 30 16:47:30.500774 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.500734    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/52a9edcf-c077-4650-844b-24a3c6a0f87e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c8f575fb-jsx25\" (UID: \"52a9edcf-c077-4650-844b-24a3c6a0f87e\") " pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25"
Oct 30 16:47:30.500774 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.500762    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/52a9edcf-c077-4650-844b-24a3c6a0f87e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c8f575fb-jsx25\" (UID: \"52a9edcf-c077-4650-844b-24a3c6a0f87e\") " pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25"
Oct 30 16:47:30.601741 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.601651    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/52a9edcf-c077-4650-844b-24a3c6a0f87e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c8f575fb-jsx25\" (UID: \"52a9edcf-c077-4650-844b-24a3c6a0f87e\") " pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25"
Oct 30 16:47:30.601741 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.601708    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnktd\" (UniqueName: \"kubernetes.io/projected/52a9edcf-c077-4650-844b-24a3c6a0f87e-kube-api-access-tnktd\") pod \"thanos-querier-7c8f575fb-jsx25\" (UID: \"52a9edcf-c077-4650-844b-24a3c6a0f87e\") " pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25"
Oct 30 16:47:30.601741 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.601737    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52a9edcf-c077-4650-844b-24a3c6a0f87e-tmp\") pod \"thanos-querier-7c8f575fb-jsx25\" (UID: \"52a9edcf-c077-4650-844b-24a3c6a0f87e\") " pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25"
Oct 30 16:47:30.602017 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.601765    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/52a9edcf-c077-4650-844b-24a3c6a0f87e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c8f575fb-jsx25\" (UID: \"52a9edcf-c077-4650-844b-24a3c6a0f87e\") " pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25"
Oct 30 16:47:30.602017 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.601799    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52a9edcf-c077-4650-844b-24a3c6a0f87e-metrics-client-ca\") pod \"thanos-querier-7c8f575fb-jsx25\" (UID: \"52a9edcf-c077-4650-844b-24a3c6a0f87e\") " pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25"
Oct 30 16:47:30.602017 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.601840    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/52a9edcf-c077-4650-844b-24a3c6a0f87e-secret-thanos-querier-tls\") pod \"thanos-querier-7c8f575fb-jsx25\" (UID: \"52a9edcf-c077-4650-844b-24a3c6a0f87e\") " pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25"
Oct 30 16:47:30.602017 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.601874    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/52a9edcf-c077-4650-844b-24a3c6a0f87e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c8f575fb-jsx25\" (UID: \"52a9edcf-c077-4650-844b-24a3c6a0f87e\") " pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25"
Oct 30 16:47:30.602017 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.601916    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/52a9edcf-c077-4650-844b-24a3c6a0f87e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c8f575fb-jsx25\" (UID: \"52a9edcf-c077-4650-844b-24a3c6a0f87e\") " pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25"
Oct 30 16:47:30.602017 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.601954    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/52a9edcf-c077-4650-844b-24a3c6a0f87e-secret-grpc-tls\") pod \"thanos-querier-7c8f575fb-jsx25\" (UID: \"52a9edcf-c077-4650-844b-24a3c6a0f87e\") " pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25"
Oct 30 16:47:30.602308 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.602246    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52a9edcf-c077-4650-844b-24a3c6a0f87e-tmp\") pod \"thanos-querier-7c8f575fb-jsx25\" (UID: \"52a9edcf-c077-4650-844b-24a3c6a0f87e\") " pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25"
Oct 30 16:47:30.602899 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.602846    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52a9edcf-c077-4650-844b-24a3c6a0f87e-metrics-client-ca\") pod \"thanos-querier-7c8f575fb-jsx25\" (UID: \"52a9edcf-c077-4650-844b-24a3c6a0f87e\") " pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25"
Oct 30 16:47:30.605127 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.605099    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/52a9edcf-c077-4650-844b-24a3c6a0f87e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c8f575fb-jsx25\" (UID: \"52a9edcf-c077-4650-844b-24a3c6a0f87e\") " pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25"
Oct 30 16:47:30.605252 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.605149    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/52a9edcf-c077-4650-844b-24a3c6a0f87e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c8f575fb-jsx25\" (UID: \"52a9edcf-c077-4650-844b-24a3c6a0f87e\") " pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25"
Oct 30 16:47:30.605357 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.605339    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/52a9edcf-c077-4650-844b-24a3c6a0f87e-secret-thanos-querier-tls\") pod \"thanos-querier-7c8f575fb-jsx25\" (UID: \"52a9edcf-c077-4650-844b-24a3c6a0f87e\") " pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25"
Oct 30 16:47:30.605768 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.605744    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/52a9edcf-c077-4650-844b-24a3c6a0f87e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c8f575fb-jsx25\" (UID: \"52a9edcf-c077-4650-844b-24a3c6a0f87e\") " pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25"
Oct 30 16:47:30.606429 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.605974    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/52a9edcf-c077-4650-844b-24a3c6a0f87e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c8f575fb-jsx25\" (UID: \"52a9edcf-c077-4650-844b-24a3c6a0f87e\") " pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25"
Oct 30 16:47:30.606523 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.606493    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/52a9edcf-c077-4650-844b-24a3c6a0f87e-secret-grpc-tls\") pod \"thanos-querier-7c8f575fb-jsx25\" (UID: \"52a9edcf-c077-4650-844b-24a3c6a0f87e\") " pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25"
Oct 30 16:47:30.627419 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.627374    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnktd\" (UniqueName: \"kubernetes.io/projected/52a9edcf-c077-4650-844b-24a3c6a0f87e-kube-api-access-tnktd\") pod \"thanos-querier-7c8f575fb-jsx25\" (UID: \"52a9edcf-c077-4650-844b-24a3c6a0f87e\") " pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25"
Oct 30 16:47:30.735211 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.735171    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25"
Oct 30 16:47:30.849874 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.849840    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cv7gw" event={"ID":"7e04ca71-89c2-4232-9da2-7034affb6a99","Type":"ContainerStarted","Data":"65d4b5e48ae41044d3a73faf27c1c5290529e0e487ae81c80a7b4df695b814ed"}
Oct 30 16:47:30.850088 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.849895    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cv7gw" event={"ID":"7e04ca71-89c2-4232-9da2-7034affb6a99","Type":"ContainerStarted","Data":"a7f4946a44dfeca7ad0074fcbf7d4487a9c7965158746883d71c98d663640c42"}
Oct 30 16:47:30.851221 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.851194    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66f44d22-9ae0-44aa-a585-cf1370bdb459","Type":"ContainerStarted","Data":"dbd4c657a6c968507fafd6096d62c9ad911bc72c8ba5919992700c5ac5934e46"}
Oct 30 16:47:30.893955 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:30.893839    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cv7gw" podStartSLOduration=1.905158372 podStartE2EDuration="2.893821686s" totalImagesPullingTime="904.804473ms" totalInitContainerRuntime="83.858841ms" isStatefulPod=true podCreationTimestamp="2026-10-30 16:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:30.892899217 +0000 UTC m=+27.327801424" watchObservedRunningTime="2026-10-30 16:47:30.893821686 +0000 UTC m=+27.328723892"
Oct 30 16:47:31.049239 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:31.049126    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7c8f575fb-jsx25"]
Oct 30 16:47:31.055992 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:31.055961    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a9edcf_c077_4650_844b_24a3c6a0f87e.slice/crio-9307c646dadae2e927de9ceac3e1c591d0c1ea3270c47e741f41eb1201bb004f WatchSource:0}: Error finding container 9307c646dadae2e927de9ceac3e1c591d0c1ea3270c47e741f41eb1201bb004f: Status 404 returned error can't find the container with id 9307c646dadae2e927de9ceac3e1c591d0c1ea3270c47e741f41eb1201bb004f
Oct 30 16:47:31.414676 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:31.413484    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kkb8g"
Oct 30 16:47:31.858806 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:31.858759    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25" event={"ID":"52a9edcf-c077-4650-844b-24a3c6a0f87e","Type":"ContainerStarted","Data":"9307c646dadae2e927de9ceac3e1c591d0c1ea3270c47e741f41eb1201bb004f"}
Oct 30 16:47:31.860906 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:31.860862    4433 generic.go:356] "Generic (PLEG): container finished" podID="66f44d22-9ae0-44aa-a585-cf1370bdb459" containerID="5fa2b6cc88ff8efaf67af4ff267914139bc2ad2ce9904a20c89167c64ee70a55" exitCode=0
Oct 30 16:47:31.862234 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:31.862209    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66f44d22-9ae0-44aa-a585-cf1370bdb459","Type":"ContainerDied","Data":"5fa2b6cc88ff8efaf67af4ff267914139bc2ad2ce9904a20c89167c64ee70a55"}
Oct 30 16:47:32.642887 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.642851    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-r2pxh"
Oct 30 16:47:32.671470 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.671435    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-854dcbf887-h7wp4"]
Oct 30 16:47:32.675436 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.675392    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-854dcbf887-h7wp4"
Oct 30 16:47:32.696765 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.696677    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dh9oovo1f30vv\""
Oct 30 16:47:32.696953 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.696898    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-j7mck\""
Oct 30 16:47:32.697022 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.696986    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 16:47:32.697022 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.697001    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 16:47:32.697140 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.697121    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 16:47:32.697140 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.696903    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 16:47:32.698375 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.698294    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-854dcbf887-h7wp4"]
Oct 30 16:47:32.822977 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.822936    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c-audit-log\") pod \"metrics-server-854dcbf887-h7wp4\" (UID: \"44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c\") " pod="openshift-monitoring/metrics-server-854dcbf887-h7wp4"
Oct 30 16:47:32.823176 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.822993    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c-client-ca-bundle\") pod \"metrics-server-854dcbf887-h7wp4\" (UID: \"44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c\") " pod="openshift-monitoring/metrics-server-854dcbf887-h7wp4"
Oct 30 16:47:32.823176 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.823025    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c-secret-metrics-server-client-certs\") pod \"metrics-server-854dcbf887-h7wp4\" (UID: \"44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c\") " pod="openshift-monitoring/metrics-server-854dcbf887-h7wp4"
Oct 30 16:47:32.823176 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.823117    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c-metrics-server-audit-profiles\") pod \"metrics-server-854dcbf887-h7wp4\" (UID: \"44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c\") " pod="openshift-monitoring/metrics-server-854dcbf887-h7wp4"
Oct 30 16:47:32.823176 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.823155    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6f47\" (UniqueName: \"kubernetes.io/projected/44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c-kube-api-access-s6f47\") pod \"metrics-server-854dcbf887-h7wp4\" (UID: \"44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c\") " pod="openshift-monitoring/metrics-server-854dcbf887-h7wp4"
Oct 30 16:47:32.823438 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.823193    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-854dcbf887-h7wp4\" (UID: \"44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c\") " pod="openshift-monitoring/metrics-server-854dcbf887-h7wp4"
Oct 30 16:47:32.823438 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.823226    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c-secret-metrics-server-tls\") pod \"metrics-server-854dcbf887-h7wp4\" (UID: \"44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c\") " pod="openshift-monitoring/metrics-server-854dcbf887-h7wp4"
Oct 30 16:47:32.926831 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.924058    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c-metrics-server-audit-profiles\") pod \"metrics-server-854dcbf887-h7wp4\" (UID: \"44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c\") " pod="openshift-monitoring/metrics-server-854dcbf887-h7wp4"
Oct 30 16:47:32.926831 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.924129    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6f47\" (UniqueName: \"kubernetes.io/projected/44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c-kube-api-access-s6f47\") pod \"metrics-server-854dcbf887-h7wp4\" (UID: \"44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c\") " pod="openshift-monitoring/metrics-server-854dcbf887-h7wp4"
Oct 30 16:47:32.926831 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.924156    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-854dcbf887-h7wp4\" (UID: \"44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c\") " pod="openshift-monitoring/metrics-server-854dcbf887-h7wp4"
Oct 30 16:47:32.926831 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.924177    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c-secret-metrics-server-tls\") pod \"metrics-server-854dcbf887-h7wp4\" (UID: \"44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c\") " pod="openshift-monitoring/metrics-server-854dcbf887-h7wp4"
Oct 30 16:47:32.926831 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.924204    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c-audit-log\") pod \"metrics-server-854dcbf887-h7wp4\" (UID: \"44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c\") " pod="openshift-monitoring/metrics-server-854dcbf887-h7wp4"
Oct 30 16:47:32.926831 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.924226    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c-client-ca-bundle\") pod \"metrics-server-854dcbf887-h7wp4\" (UID: \"44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c\") " pod="openshift-monitoring/metrics-server-854dcbf887-h7wp4"
Oct 30 16:47:32.926831 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.924243    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c-secret-metrics-server-client-certs\") pod \"metrics-server-854dcbf887-h7wp4\" (UID: \"44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c\") " pod="openshift-monitoring/metrics-server-854dcbf887-h7wp4"
Oct 30 16:47:32.926831 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.925268    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-854dcbf887-h7wp4\" (UID: \"44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c\") " pod="openshift-monitoring/metrics-server-854dcbf887-h7wp4"
Oct 30 16:47:32.926831 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.925958    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c-metrics-server-audit-profiles\") pod \"metrics-server-854dcbf887-h7wp4\" (UID: \"44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c\") " pod="openshift-monitoring/metrics-server-854dcbf887-h7wp4"
Oct 30 16:47:32.926831 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.926517    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c-audit-log\") pod \"metrics-server-854dcbf887-h7wp4\" (UID: \"44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c\") " pod="openshift-monitoring/metrics-server-854dcbf887-h7wp4"
Oct 30 16:47:32.930505 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.930475    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c-secret-metrics-server-tls\") pod \"metrics-server-854dcbf887-h7wp4\" (UID: \"44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c\") " pod="openshift-monitoring/metrics-server-854dcbf887-h7wp4"
Oct 30 16:47:32.931085 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.931061    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c-secret-metrics-server-client-certs\") pod \"metrics-server-854dcbf887-h7wp4\" (UID: \"44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c\") " pod="openshift-monitoring/metrics-server-854dcbf887-h7wp4"
Oct 30 16:47:32.935799 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.935772    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c-client-ca-bundle\") pod \"metrics-server-854dcbf887-h7wp4\" (UID: \"44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c\") " pod="openshift-monitoring/metrics-server-854dcbf887-h7wp4"
Oct 30 16:47:32.942864 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.942829    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-bbb4g"]
Oct 30 16:47:32.947321 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.947291    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bbb4g"
Oct 30 16:47:32.954092 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.954053    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6f47\" (UniqueName: \"kubernetes.io/projected/44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c-kube-api-access-s6f47\") pod \"metrics-server-854dcbf887-h7wp4\" (UID: \"44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c\") " pod="openshift-monitoring/metrics-server-854dcbf887-h7wp4"
Oct 30 16:47:32.969263 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.969226    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 16:47:32.969448 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.969424    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-bbb4g"]
Oct 30 16:47:32.979941 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.979903    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-p25gg\""
Oct 30 16:47:32.991602 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:32.991568    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-854dcbf887-h7wp4"
Oct 30 16:47:33.126128 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:33.126090    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/215e894c-1ace-4564-a694-d535a1cffeba-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-bbb4g\" (UID: \"215e894c-1ace-4564-a694-d535a1cffeba\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bbb4g"
Oct 30 16:47:33.126305 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:33.126146    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw5gp\" (UniqueName: \"kubernetes.io/projected/215e894c-1ace-4564-a694-d535a1cffeba-kube-api-access-xw5gp\") pod \"monitoring-plugin-77f4f76594-bbb4g\" (UID: \"215e894c-1ace-4564-a694-d535a1cffeba\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bbb4g"
Oct 30 16:47:33.226774 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:33.226689    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/215e894c-1ace-4564-a694-d535a1cffeba-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-bbb4g\" (UID: \"215e894c-1ace-4564-a694-d535a1cffeba\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bbb4g"
Oct 30 16:47:33.226774 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:33.226750    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw5gp\" (UniqueName: \"kubernetes.io/projected/215e894c-1ace-4564-a694-d535a1cffeba-kube-api-access-xw5gp\") pod \"monitoring-plugin-77f4f76594-bbb4g\" (UID: \"215e894c-1ace-4564-a694-d535a1cffeba\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bbb4g"
Oct 30 16:47:33.229638 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:33.229585    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/215e894c-1ace-4564-a694-d535a1cffeba-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-bbb4g\" (UID: \"215e894c-1ace-4564-a694-d535a1cffeba\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bbb4g"
Oct 30 16:47:33.254527 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:33.254490    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw5gp\" (UniqueName: \"kubernetes.io/projected/215e894c-1ace-4564-a694-d535a1cffeba-kube-api-access-xw5gp\") pod \"monitoring-plugin-77f4f76594-bbb4g\" (UID: \"215e894c-1ace-4564-a694-d535a1cffeba\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bbb4g"
Oct 30 16:47:33.269251 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:33.269213    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bbb4g"
Oct 30 16:47:33.316660 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:33.316376    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-854dcbf887-h7wp4"]
Oct 30 16:47:33.324432 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:33.324386    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44f1b4ab_6c59_47e3_9c20_d3c6cda4f21c.slice/crio-d90e5e348f18a4899bedc53e565151e1161b567adfd35f0109ca1424d1371529 WatchSource:0}: Error finding container d90e5e348f18a4899bedc53e565151e1161b567adfd35f0109ca1424d1371529: Status 404 returned error can't find the container with id d90e5e348f18a4899bedc53e565151e1161b567adfd35f0109ca1424d1371529
Oct 30 16:47:33.582219 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:33.582195    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-bbb4g"]
Oct 30 16:47:33.699668 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:33.699012    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vw5v7"
Oct 30 16:47:33.886431 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:33.886372    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25" event={"ID":"52a9edcf-c077-4650-844b-24a3c6a0f87e","Type":"ContainerStarted","Data":"98d8cd7bf4f88976aa73b3b9f06cf560d83b609615595f2bd3abd268453690e9"}
Oct 30 16:47:33.886729 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:33.886706    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25" event={"ID":"52a9edcf-c077-4650-844b-24a3c6a0f87e","Type":"ContainerStarted","Data":"f0d0249fa702c87cc6d30ea56872f6c29817bdfdc88a3537b42c2f5432275d9a"}
Oct 30 16:47:33.887136 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:33.887112    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25" event={"ID":"52a9edcf-c077-4650-844b-24a3c6a0f87e","Type":"ContainerStarted","Data":"8755006c1d688c5e60527c1da4099babc56e8748f12d0e732cd4183d5f4bf195"}
Oct 30 16:47:33.895668 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:33.895635    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66f44d22-9ae0-44aa-a585-cf1370bdb459","Type":"ContainerStarted","Data":"00d377b51fed238ae4e97d306111b7238f08fd4410c737d3a45aad8dd2b01a94"}
Oct 30 16:47:33.895769 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:33.895715    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66f44d22-9ae0-44aa-a585-cf1370bdb459","Type":"ContainerStarted","Data":"c3e0e529d46e7aa7e45bf451750259a576157961cfb41c02df2256229cf388a4"}
Oct 30 16:47:33.895769 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:33.895758    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66f44d22-9ae0-44aa-a585-cf1370bdb459","Type":"ContainerStarted","Data":"c2ca6f96075b549ce738651953f86e0c606fe3ee92a81be204269b727bec787a"}
Oct 30 16:47:33.897667 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:33.897637    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bbb4g" event={"ID":"215e894c-1ace-4564-a694-d535a1cffeba","Type":"ContainerStarted","Data":"bcb0052ae358077b3b2919def3e88164694f40e5fcd0320f5312a3cf864acec2"}
Oct 30 16:47:33.899179 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:33.899152    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-854dcbf887-h7wp4" event={"ID":"44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c","Type":"ContainerStarted","Data":"d90e5e348f18a4899bedc53e565151e1161b567adfd35f0109ca1424d1371529"}
Oct 30 16:47:34.823438 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:34.822783    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5ztj2"
Oct 30 16:47:34.908222 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:34.908159    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66f44d22-9ae0-44aa-a585-cf1370bdb459","Type":"ContainerStarted","Data":"481f1c7f138c9eddcb0ab4bc4b2b1b6fc51a886eb2aeed11419dbaba9707ad75"}
Oct 30 16:47:34.908447 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:34.908299    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66f44d22-9ae0-44aa-a585-cf1370bdb459","Type":"ContainerStarted","Data":"f259c4a4af60b21973bcc5de4c39a2b7a7e53616d2af249a91655d87f4594a3c"}
Oct 30 16:47:35.220503 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.220467    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 16:47:35.232937 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.232865    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.249435 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.248160    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 16:47:35.267245 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.266912    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 16:47:35.267245 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.266993    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 16:47:35.267245 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.267011    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 16:47:35.267245 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.267038    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 16:47:35.267245 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.267068    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 16:47:35.267245 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.267086    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 16:47:35.267245 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.267124    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 16:47:35.267245 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.267131    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 16:47:35.267245 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.267199    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rgj9h\""
Oct 30 16:47:35.267245 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.267198    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 16:47:35.267245 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.267221    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 16:47:35.290580 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.290552    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 16:47:35.290729 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.290697    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 16:47:35.346927 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.346880    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-config\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.346927 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.346924    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0dcced64-e20a-4247-aa7d-314a1e2de54a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.347174 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.346948    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0dcced64-e20a-4247-aa7d-314a1e2de54a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.347174 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.347025    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.347174 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.347136    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.347320 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.347172    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.347320 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.347205    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.347320 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.347291    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-web-config\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.347519 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.347340    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.347519 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.347373    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.347519 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.347439    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.347519 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.347489    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.347665 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.347523    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8pln\" (UniqueName: \"kubernetes.io/projected/0dcced64-e20a-4247-aa7d-314a1e2de54a-kube-api-access-j8pln\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.347665 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.347556    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0dcced64-e20a-4247-aa7d-314a1e2de54a-config-out\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.347665 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.347592    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.347665 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.347623    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.347786 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.347664    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.347786 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.347697    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.347786 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.347728    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.347786 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.347771    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.354471 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.354363    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 16:47:35.354634 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.354598    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6ig53cld0vd0\""
Oct 30 16:47:35.354688 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.354643    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 16:47:35.448665 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.448623    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.448837 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.448675    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.448837 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.448702    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.448837 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.448734    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.448837 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.448760    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8pln\" (UniqueName: \"kubernetes.io/projected/0dcced64-e20a-4247-aa7d-314a1e2de54a-kube-api-access-j8pln\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.448837 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.448788    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0dcced64-e20a-4247-aa7d-314a1e2de54a-config-out\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.448837 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.448818    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.449095 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.448849    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.449095 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.448877    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.449095 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.448906    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.449095 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.448935    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.449095 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.448972    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.449095 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.449028    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-config\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.449095 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.449049    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0dcced64-e20a-4247-aa7d-314a1e2de54a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.449095 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.449081    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0dcced64-e20a-4247-aa7d-314a1e2de54a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.449497 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.449107    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.449497 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.449134    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.449497 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.449162    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.449497 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.449189    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.449497 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.449221    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-web-config\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.450116 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.450092    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0dcced64-e20a-4247-aa7d-314a1e2de54a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.451765 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.451740    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.452325 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.452303    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.454036 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.453118    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.454036 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.453440    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.454036 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.453960    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.454036 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.454008    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.454240 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.454039    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.456142 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.456114    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.456299 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.456247    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.456715 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.456673    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0dcced64-e20a-4247-aa7d-314a1e2de54a-config-out\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.458746 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.458701    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.465734 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.465705    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-config\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.466153 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.466132    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.466539 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.466519    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0dcced64-e20a-4247-aa7d-314a1e2de54a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.467030 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.466979    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.468767 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.468742    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.469382 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.469348    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-web-config\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.471166 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.471108    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.476712 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.476687    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8pln\" (UniqueName: \"kubernetes.io/projected/0dcced64-e20a-4247-aa7d-314a1e2de54a-kube-api-access-j8pln\") pod \"prometheus-k8s-0\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.618205 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.618163    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-597f9b54b6-nhbr9"
Oct 30 16:47:35.618955 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.618929    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-597f9b54b6-nhbr9"
Oct 30 16:47:35.624086 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.624063    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-597f9b54b6-nhbr9"
Oct 30 16:47:35.687941 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.687696    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:35.916756 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.916717    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25" event={"ID":"52a9edcf-c077-4650-844b-24a3c6a0f87e","Type":"ContainerStarted","Data":"950351f8466fa64a2f55bd41eb6406ca673bfdd7fd48757cf82fae588e6d8445"}
Oct 30 16:47:35.917156 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.916812    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25" event={"ID":"52a9edcf-c077-4650-844b-24a3c6a0f87e","Type":"ContainerStarted","Data":"ac84c47258a4d033d5c551eb17f4755ebc965a95d1d98f586dfaf1ac3fcd72a7"}
Oct 30 16:47:35.924979 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.924880    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66f44d22-9ae0-44aa-a585-cf1370bdb459","Type":"ContainerStarted","Data":"51dc261cc66033589c0d98b13207212d0731d2d4767e87a78d2555057a542bb9"}
Oct 30 16:47:35.929061 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.929036    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-597f9b54b6-nhbr9"
Oct 30 16:47:35.987830 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:35.981170    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.620299412 podStartE2EDuration="6.981155927s" totalImagesPullingTime="4.347181528s" totalInitContainerRuntime="13.674987ms" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:35.980523797 +0000 UTC m=+32.415426007" watchObservedRunningTime="2026-10-30 16:47:35.981155927 +0000 UTC m=+32.416058133"
Oct 30 16:47:36.249061 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:36.249012    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 16:47:36.396439 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:36.396378    4433 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dcced64_e20a_4247_aa7d_314a1e2de54a.slice/crio-45fd61c193da51e770f95894b7fcc5cd41594c13f8db715997019199eb8d57ce.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 16:47:36.932566 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:36.932526    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25" event={"ID":"52a9edcf-c077-4650-844b-24a3c6a0f87e","Type":"ContainerStarted","Data":"15ea3c35d309911b6296cd540777a2795c4eef1d3eb5f73dde49b67be37735a9"}
Oct 30 16:47:36.933048 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:36.932674    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25"
Oct 30 16:47:36.934649 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:36.934623    4433 generic.go:356] "Generic (PLEG): container finished" podID="0dcced64-e20a-4247-aa7d-314a1e2de54a" containerID="45fd61c193da51e770f95894b7fcc5cd41594c13f8db715997019199eb8d57ce" exitCode=0
Oct 30 16:47:36.934781 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:36.934706    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0dcced64-e20a-4247-aa7d-314a1e2de54a","Type":"ContainerDied","Data":"45fd61c193da51e770f95894b7fcc5cd41594c13f8db715997019199eb8d57ce"}
Oct 30 16:47:36.934835 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:36.934783    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0dcced64-e20a-4247-aa7d-314a1e2de54a","Type":"ContainerStarted","Data":"010a16365b187fcd2ab284a87d3277005a7cc790f031ca2b3e56e392d93e17a7"}
Oct 30 16:47:36.936500 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:36.936472    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bbb4g" event={"ID":"215e894c-1ace-4564-a694-d535a1cffeba","Type":"ContainerStarted","Data":"eb5987e8269f3fe870a054cc5fe75e3e7c8ae7d0dcd896c92ede7fcbd99e2c10"}
Oct 30 16:47:36.936939 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:36.936920    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bbb4g"
Oct 30 16:47:36.938491 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:36.938465    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-854dcbf887-h7wp4" event={"ID":"44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c","Type":"ContainerStarted","Data":"f86cf5306cc6b5a5efba31b41ebde843b18cfd2aa1d9e91eb5f091b9854c55a3"}
Oct 30 16:47:36.942916 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:36.942897    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bbb4g"
Oct 30 16:47:36.983740 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:36.983667    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25" podStartSLOduration=3.072365635 podStartE2EDuration="6.983648921s" totalImagesPullingTime="3.911283286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:36.981426859 +0000 UTC m=+33.416329065" watchObservedRunningTime="2026-10-30 16:47:36.983648921 +0000 UTC m=+33.418551130"
Oct 30 16:47:37.024617 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:37.024531    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-854dcbf887-h7wp4" podStartSLOduration=2.433111841 podStartE2EDuration="5.024512425s" totalImagesPullingTime="2.591400584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:37.023466974 +0000 UTC m=+33.458369181" watchObservedRunningTime="2026-10-30 16:47:37.024512425 +0000 UTC m=+33.459414632"
Oct 30 16:47:37.126268 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:37.126212    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bbb4g" podStartSLOduration=2.567831167 podStartE2EDuration="5.126194319s" totalImagesPullingTime="2.558363152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:37.12529405 +0000 UTC m=+33.560196255" watchObservedRunningTime="2026-10-30 16:47:37.126194319 +0000 UTC m=+33.561096524"
Oct 30 16:47:37.737223 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:37.737187    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5584db78d6-2wdm4"]
Oct 30 16:47:37.755488 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:37.755035    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5584db78d6-2wdm4"
Oct 30 16:47:37.763210 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:37.763177    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5584db78d6-2wdm4"]
Oct 30 16:47:37.777974 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:37.777890    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 16:47:37.878606 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:37.878567    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/914e3790-ef93-47ee-90e9-419cc4a7f3ca-tmp\") pod \"console-5584db78d6-2wdm4\" (UID: \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\") " pod="openshift-console/console-5584db78d6-2wdm4"
Oct 30 16:47:37.878783 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:37.878637    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/914e3790-ef93-47ee-90e9-419cc4a7f3ca-service-ca\") pod \"console-5584db78d6-2wdm4\" (UID: \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\") " pod="openshift-console/console-5584db78d6-2wdm4"
Oct 30 16:47:37.878783 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:37.878680    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/914e3790-ef93-47ee-90e9-419cc4a7f3ca-console-oauth-config\") pod \"console-5584db78d6-2wdm4\" (UID: \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\") " pod="openshift-console/console-5584db78d6-2wdm4"
Oct 30 16:47:37.878783 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:37.878711    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/914e3790-ef93-47ee-90e9-419cc4a7f3ca-oauth-serving-cert\") pod \"console-5584db78d6-2wdm4\" (UID: \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\") " pod="openshift-console/console-5584db78d6-2wdm4"
Oct 30 16:47:37.878783 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:37.878744    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/914e3790-ef93-47ee-90e9-419cc4a7f3ca-trusted-ca-bundle\") pod \"console-5584db78d6-2wdm4\" (UID: \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\") " pod="openshift-console/console-5584db78d6-2wdm4"
Oct 30 16:47:37.878933 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:37.878789    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75pps\" (UniqueName: \"kubernetes.io/projected/914e3790-ef93-47ee-90e9-419cc4a7f3ca-kube-api-access-75pps\") pod \"console-5584db78d6-2wdm4\" (UID: \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\") " pod="openshift-console/console-5584db78d6-2wdm4"
Oct 30 16:47:37.878933 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:37.878819    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/914e3790-ef93-47ee-90e9-419cc4a7f3ca-console-serving-cert\") pod \"console-5584db78d6-2wdm4\" (UID: \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\") " pod="openshift-console/console-5584db78d6-2wdm4"
Oct 30 16:47:37.878933 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:37.878852    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/914e3790-ef93-47ee-90e9-419cc4a7f3ca-console-config\") pod \"console-5584db78d6-2wdm4\" (UID: \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\") " pod="openshift-console/console-5584db78d6-2wdm4"
Oct 30 16:47:37.981385 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:37.980041    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/914e3790-ef93-47ee-90e9-419cc4a7f3ca-tmp\") pod \"console-5584db78d6-2wdm4\" (UID: \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\") " pod="openshift-console/console-5584db78d6-2wdm4"
Oct 30 16:47:37.981385 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:37.980540    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/914e3790-ef93-47ee-90e9-419cc4a7f3ca-service-ca\") pod \"console-5584db78d6-2wdm4\" (UID: \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\") " pod="openshift-console/console-5584db78d6-2wdm4"
Oct 30 16:47:37.981385 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:37.980584    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/914e3790-ef93-47ee-90e9-419cc4a7f3ca-console-oauth-config\") pod \"console-5584db78d6-2wdm4\" (UID: \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\") " pod="openshift-console/console-5584db78d6-2wdm4"
Oct 30 16:47:37.981385 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:37.980613    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/914e3790-ef93-47ee-90e9-419cc4a7f3ca-oauth-serving-cert\") pod \"console-5584db78d6-2wdm4\" (UID: \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\") " pod="openshift-console/console-5584db78d6-2wdm4"
Oct 30 16:47:37.981385 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:37.980644    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/914e3790-ef93-47ee-90e9-419cc4a7f3ca-trusted-ca-bundle\") pod \"console-5584db78d6-2wdm4\" (UID: \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\") " pod="openshift-console/console-5584db78d6-2wdm4"
Oct 30 16:47:37.981385 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:37.980691    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75pps\" (UniqueName: \"kubernetes.io/projected/914e3790-ef93-47ee-90e9-419cc4a7f3ca-kube-api-access-75pps\") pod \"console-5584db78d6-2wdm4\" (UID: \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\") " pod="openshift-console/console-5584db78d6-2wdm4"
Oct 30 16:47:37.981385 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:37.980718    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/914e3790-ef93-47ee-90e9-419cc4a7f3ca-console-serving-cert\") pod \"console-5584db78d6-2wdm4\" (UID: \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\") " pod="openshift-console/console-5584db78d6-2wdm4"
Oct 30 16:47:37.981385 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:37.980761    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/914e3790-ef93-47ee-90e9-419cc4a7f3ca-console-config\") pod \"console-5584db78d6-2wdm4\" (UID: \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\") " pod="openshift-console/console-5584db78d6-2wdm4"
Oct 30 16:47:37.981385 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:37.981350    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/914e3790-ef93-47ee-90e9-419cc4a7f3ca-console-config\") pod \"console-5584db78d6-2wdm4\" (UID: \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\") " pod="openshift-console/console-5584db78d6-2wdm4"
Oct 30 16:47:37.981385 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:37.980458    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/914e3790-ef93-47ee-90e9-419cc4a7f3ca-tmp\") pod \"console-5584db78d6-2wdm4\" (UID: \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\") " pod="openshift-console/console-5584db78d6-2wdm4"
Oct 30 16:47:37.982228 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:37.981890    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/914e3790-ef93-47ee-90e9-419cc4a7f3ca-service-ca\") pod \"console-5584db78d6-2wdm4\" (UID: \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\") " pod="openshift-console/console-5584db78d6-2wdm4"
Oct 30 16:47:37.983110 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:37.983083    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/914e3790-ef93-47ee-90e9-419cc4a7f3ca-trusted-ca-bundle\") pod \"console-5584db78d6-2wdm4\" (UID: \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\") " pod="openshift-console/console-5584db78d6-2wdm4"
Oct 30 16:47:37.983731 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:37.983709    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/914e3790-ef93-47ee-90e9-419cc4a7f3ca-oauth-serving-cert\") pod \"console-5584db78d6-2wdm4\" (UID: \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\") " pod="openshift-console/console-5584db78d6-2wdm4"
Oct 30 16:47:37.984883 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:37.984859    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/914e3790-ef93-47ee-90e9-419cc4a7f3ca-console-oauth-config\") pod \"console-5584db78d6-2wdm4\" (UID: \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\") " pod="openshift-console/console-5584db78d6-2wdm4"
Oct 30 16:47:37.986723 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:37.986697    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/914e3790-ef93-47ee-90e9-419cc4a7f3ca-console-serving-cert\") pod \"console-5584db78d6-2wdm4\" (UID: \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\") " pod="openshift-console/console-5584db78d6-2wdm4"
Oct 30 16:47:38.011564 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:38.011497    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75pps\" (UniqueName: \"kubernetes.io/projected/914e3790-ef93-47ee-90e9-419cc4a7f3ca-kube-api-access-75pps\") pod \"console-5584db78d6-2wdm4\" (UID: \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\") " pod="openshift-console/console-5584db78d6-2wdm4"
Oct 30 16:47:38.097698 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:38.097657    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5584db78d6-2wdm4"
Oct 30 16:47:38.447013 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:38.446970    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5584db78d6-2wdm4"]
Oct 30 16:47:38.453589 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:47:38.453552    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod914e3790_ef93_47ee_90e9_419cc4a7f3ca.slice/crio-42233afa3cac1f09786ffedb28ef27ab35043df29c8eb6a4ea22819f2f4e14c4 WatchSource:0}: Error finding container 42233afa3cac1f09786ffedb28ef27ab35043df29c8eb6a4ea22819f2f4e14c4: Status 404 returned error can't find the container with id 42233afa3cac1f09786ffedb28ef27ab35043df29c8eb6a4ea22819f2f4e14c4
Oct 30 16:47:38.950033 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:38.949764    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5584db78d6-2wdm4" event={"ID":"914e3790-ef93-47ee-90e9-419cc4a7f3ca","Type":"ContainerStarted","Data":"13f18921aa9bad120c55a357c550659ab8b71f5734a25ad6bcd74896c371f62a"}
Oct 30 16:47:38.950033 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:38.949824    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5584db78d6-2wdm4" event={"ID":"914e3790-ef93-47ee-90e9-419cc4a7f3ca","Type":"ContainerStarted","Data":"42233afa3cac1f09786ffedb28ef27ab35043df29c8eb6a4ea22819f2f4e14c4"}
Oct 30 16:47:38.998703 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:38.998650    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5584db78d6-2wdm4" podStartSLOduration=1.9986291 podStartE2EDuration="1.9986291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:38.995451613 +0000 UTC m=+35.430353818" watchObservedRunningTime="2026-10-30 16:47:38.9986291 +0000 UTC m=+35.433531312"
Oct 30 16:47:40.966914 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:40.966872    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0dcced64-e20a-4247-aa7d-314a1e2de54a","Type":"ContainerStarted","Data":"1f629cbf1c6bc48868eff9832329db0ba715fb878d2ea1b62b4b1f49448df1e3"}
Oct 30 16:47:40.967397 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:40.966963    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0dcced64-e20a-4247-aa7d-314a1e2de54a","Type":"ContainerStarted","Data":"d482751a418acea4811f69cfe03140b0c77049c0db4a4c8f59c2f8e0acecad78"}
Oct 30 16:47:40.967397 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:40.967005    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0dcced64-e20a-4247-aa7d-314a1e2de54a","Type":"ContainerStarted","Data":"09297173c46029a0dca8195cac8af92f1dcc3d1c1fb8490d75361abe48921af3"}
Oct 30 16:47:40.970498 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:40.970225    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-rtst9" event={"ID":"dd9cd8fd-063f-4f87-a80c-57f65f0d220b","Type":"ContainerStarted","Data":"07b4d16e4e70237c0fc58ba29416d231db6c706b26b57a5777b0c7b5db12bf9b"}
Oct 30 16:47:40.971724 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:40.971386    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-rtst9"
Oct 30 16:47:40.988434 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:40.985230    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-rtst9"
Oct 30 16:47:41.081658 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:41.080827    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-rtst9" podStartSLOduration=1.736901432 podStartE2EDuration="21.080807664s" totalImagesPullingTime="19.343906232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:41.037669097 +0000 UTC m=+37.472571306" watchObservedRunningTime="2026-10-30 16:47:41.080807664 +0000 UTC m=+37.515709869"
Oct 30 16:47:41.979361 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:41.979317    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0dcced64-e20a-4247-aa7d-314a1e2de54a","Type":"ContainerStarted","Data":"35e77c7bf4dd7af16a9cd4fd165e0c61e2920ca2efbd1b51d08f3c7faefaa416"}
Oct 30 16:47:41.979989 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:41.979433    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0dcced64-e20a-4247-aa7d-314a1e2de54a","Type":"ContainerStarted","Data":"7b75b3baa9832ef9cae5c4a477577449f821a3fa40e67f12a1c641a22096e95c"}
Oct 30 16:47:41.979989 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:41.979486    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0dcced64-e20a-4247-aa7d-314a1e2de54a","Type":"ContainerStarted","Data":"114dcb058dd2d13daf31e393642b8b3b66d0c45c62ddf482a476c9e71c7b8ca1"}
Oct 30 16:47:42.072107 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:42.071968    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.289576147 podStartE2EDuration="7.071946746s" totalImagesPullingTime="3.755694874s" totalInitContainerRuntime="26.675725ms" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:42.059446861 +0000 UTC m=+38.494349062" watchObservedRunningTime="2026-10-30 16:47:42.071946746 +0000 UTC m=+38.506848953"
Oct 30 16:47:42.948759 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:42.948727    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7c8f575fb-jsx25"
Oct 30 16:47:45.693097 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:45.693064    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:48.097929 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:48.097880    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5584db78d6-2wdm4"
Oct 30 16:47:48.098549 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:48.097987    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5584db78d6-2wdm4"
Oct 30 16:47:48.104739 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:48.104715    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5584db78d6-2wdm4"
Oct 30 16:47:49.013706 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:49.013677    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5584db78d6-2wdm4"
Oct 30 16:47:49.089823 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:49.089794    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-597f9b54b6-nhbr9"]
Oct 30 16:47:52.991952 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:52.991917    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-854dcbf887-h7wp4"
Oct 30 16:47:52.992380 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:52.992003    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-854dcbf887-h7wp4"
Oct 30 16:47:56.229745 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:56.229704    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:56.279631 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:56.279605    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:47:56.977035 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:47:56.976994    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 16:47:57.059156 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:47:57.059128    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:12.997110 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:12.997078    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-854dcbf887-h7wp4"
Oct 30 16:48:13.000739 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:13.000717    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-854dcbf887-h7wp4"
Oct 30 16:48:14.111468 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:14.111383    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-597f9b54b6-nhbr9" podUID="5a9be69d-b840-4887-8a9a-822879938df1" containerName="console" containerID="cri-o://6df394d9d4f038b4014ba4af6934d6e06f4cd283f17da6b8edbd1a07218ad8f3" gracePeriod=15
Oct 30 16:48:14.394583 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:14.394552    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-597f9b54b6-nhbr9"
Oct 30 16:48:14.537026 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:14.536994    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a9be69d-b840-4887-8a9a-822879938df1-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a9be69d-b840-4887-8a9a-822879938df1-console-oauth-config\") pod \"5a9be69d-b840-4887-8a9a-822879938df1\" (UID: \"5a9be69d-b840-4887-8a9a-822879938df1\") "
Oct 30 16:48:14.537216 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:14.537040    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9be69d-b840-4887-8a9a-822879938df1-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a9be69d-b840-4887-8a9a-822879938df1-tmp\") pod \"5a9be69d-b840-4887-8a9a-822879938df1\" (UID: \"5a9be69d-b840-4887-8a9a-822879938df1\") "
Oct 30 16:48:14.537216 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:14.537060    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a9be69d-b840-4887-8a9a-822879938df1-kube-api-access-28h9w\" (UniqueName: \"kubernetes.io/projected/5a9be69d-b840-4887-8a9a-822879938df1-kube-api-access-28h9w\") pod \"5a9be69d-b840-4887-8a9a-822879938df1\" (UID: \"5a9be69d-b840-4887-8a9a-822879938df1\") "
Oct 30 16:48:14.537216 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:14.537098    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a9be69d-b840-4887-8a9a-822879938df1-console-config\" (UniqueName: \"kubernetes.io/configmap/5a9be69d-b840-4887-8a9a-822879938df1-console-config\") pod \"5a9be69d-b840-4887-8a9a-822879938df1\" (UID: \"5a9be69d-b840-4887-8a9a-822879938df1\") "
Oct 30 16:48:14.537216 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:14.537121    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a9be69d-b840-4887-8a9a-822879938df1-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a9be69d-b840-4887-8a9a-822879938df1-oauth-serving-cert\") pod \"5a9be69d-b840-4887-8a9a-822879938df1\" (UID: \"5a9be69d-b840-4887-8a9a-822879938df1\") "
Oct 30 16:48:14.537216 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:14.537175    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a9be69d-b840-4887-8a9a-822879938df1-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a9be69d-b840-4887-8a9a-822879938df1-console-serving-cert\") pod \"5a9be69d-b840-4887-8a9a-822879938df1\" (UID: \"5a9be69d-b840-4887-8a9a-822879938df1\") "
Oct 30 16:48:14.537519 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:14.537286    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a9be69d-b840-4887-8a9a-822879938df1-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a9be69d-b840-4887-8a9a-822879938df1-service-ca\") pod \"5a9be69d-b840-4887-8a9a-822879938df1\" (UID: \"5a9be69d-b840-4887-8a9a-822879938df1\") "
Oct 30 16:48:14.537519 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:14.537313    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9be69d-b840-4887-8a9a-822879938df1-tmp" pod "5a9be69d-b840-4887-8a9a-822879938df1" (UID: "5a9be69d-b840-4887-8a9a-822879938df1"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:48:14.537639 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:14.537624    4433 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a9be69d-b840-4887-8a9a-822879938df1-tmp\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:14.537695 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:14.537659    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a9be69d-b840-4887-8a9a-822879938df1-console-config" pod "5a9be69d-b840-4887-8a9a-822879938df1" (UID: "5a9be69d-b840-4887-8a9a-822879938df1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:48:14.537695 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:14.537668    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a9be69d-b840-4887-8a9a-822879938df1-oauth-serving-cert" pod "5a9be69d-b840-4887-8a9a-822879938df1" (UID: "5a9be69d-b840-4887-8a9a-822879938df1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:48:14.537798 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:14.537705    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a9be69d-b840-4887-8a9a-822879938df1-service-ca" pod "5a9be69d-b840-4887-8a9a-822879938df1" (UID: "5a9be69d-b840-4887-8a9a-822879938df1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:48:14.539419 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:14.539372    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9be69d-b840-4887-8a9a-822879938df1-kube-api-access-28h9w" pod "5a9be69d-b840-4887-8a9a-822879938df1" (UID: "5a9be69d-b840-4887-8a9a-822879938df1"). InnerVolumeSpecName "kube-api-access-28h9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:48:14.539531 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:14.539421    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a9be69d-b840-4887-8a9a-822879938df1-console-serving-cert" pod "5a9be69d-b840-4887-8a9a-822879938df1" (UID: "5a9be69d-b840-4887-8a9a-822879938df1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:48:14.539531 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:14.539430    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a9be69d-b840-4887-8a9a-822879938df1-console-oauth-config" pod "5a9be69d-b840-4887-8a9a-822879938df1" (UID: "5a9be69d-b840-4887-8a9a-822879938df1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:48:14.638963 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:14.638878    4433 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a9be69d-b840-4887-8a9a-822879938df1-console-oauth-config\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:14.638963 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:14.638909    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28h9w\" (UniqueName: \"kubernetes.io/projected/5a9be69d-b840-4887-8a9a-822879938df1-kube-api-access-28h9w\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:14.638963 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:14.638920    4433 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a9be69d-b840-4887-8a9a-822879938df1-console-config\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:14.638963 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:14.638928    4433 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a9be69d-b840-4887-8a9a-822879938df1-oauth-serving-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:14.638963 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:14.638941    4433 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a9be69d-b840-4887-8a9a-822879938df1-console-serving-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:14.638963 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:14.638954    4433 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a9be69d-b840-4887-8a9a-822879938df1-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:15.112746 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:15.112711    4433 generic.go:356] "Generic (PLEG): container finished" podID="5a9be69d-b840-4887-8a9a-822879938df1" containerID="6df394d9d4f038b4014ba4af6934d6e06f4cd283f17da6b8edbd1a07218ad8f3" exitCode=0
Oct 30 16:48:15.113204 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:15.112760    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-597f9b54b6-nhbr9" event={"ID":"5a9be69d-b840-4887-8a9a-822879938df1","Type":"ContainerDied","Data":"6df394d9d4f038b4014ba4af6934d6e06f4cd283f17da6b8edbd1a07218ad8f3"}
Oct 30 16:48:15.113204 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:15.112764    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-597f9b54b6-nhbr9"
Oct 30 16:48:15.113204 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:15.112803    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-597f9b54b6-nhbr9" event={"ID":"5a9be69d-b840-4887-8a9a-822879938df1","Type":"ContainerDied","Data":"93194e6ef684e37f5b7fa4d9c9a9bfe71204b2e91ad7494a1eea4df5b6447c27"}
Oct 30 16:48:15.113204 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:15.112825    4433 scope.go:122] "RemoveContainer" containerID="6df394d9d4f038b4014ba4af6934d6e06f4cd283f17da6b8edbd1a07218ad8f3"
Oct 30 16:48:15.138318 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:15.138293    4433 scope.go:122] "RemoveContainer" containerID="6df394d9d4f038b4014ba4af6934d6e06f4cd283f17da6b8edbd1a07218ad8f3"
Oct 30 16:48:15.138734 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:48:15.138703    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6df394d9d4f038b4014ba4af6934d6e06f4cd283f17da6b8edbd1a07218ad8f3\": container with ID starting with 6df394d9d4f038b4014ba4af6934d6e06f4cd283f17da6b8edbd1a07218ad8f3 not found: ID does not exist" containerID="6df394d9d4f038b4014ba4af6934d6e06f4cd283f17da6b8edbd1a07218ad8f3"
Oct 30 16:48:15.138793 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:15.138744    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6df394d9d4f038b4014ba4af6934d6e06f4cd283f17da6b8edbd1a07218ad8f3"} err="failed to get container status \"6df394d9d4f038b4014ba4af6934d6e06f4cd283f17da6b8edbd1a07218ad8f3\": rpc error: code = NotFound desc = could not find container \"6df394d9d4f038b4014ba4af6934d6e06f4cd283f17da6b8edbd1a07218ad8f3\": container with ID starting with 6df394d9d4f038b4014ba4af6934d6e06f4cd283f17da6b8edbd1a07218ad8f3 not found: ID does not exist"
Oct 30 16:48:15.161532 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:15.161500    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-597f9b54b6-nhbr9"]
Oct 30 16:48:15.172610 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:15.172583    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-597f9b54b6-nhbr9"]
Oct 30 16:48:15.693165 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:15.693132    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a9be69d-b840-4887-8a9a-822879938df1" path="/var/lib/kubelet/pods/5a9be69d-b840-4887-8a9a-822879938df1/volumes"
Oct 30 16:48:19.036351 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:19.036316    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 16:48:19.036882 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:19.036645    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="66f44d22-9ae0-44aa-a585-cf1370bdb459" containerName="alertmanager" containerID="cri-o://c2ca6f96075b549ce738651953f86e0c606fe3ee92a81be204269b727bec787a" gracePeriod=120
Oct 30 16:48:19.036882 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:19.036727    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="66f44d22-9ae0-44aa-a585-cf1370bdb459" containerName="kube-rbac-proxy-metric" containerID="cri-o://481f1c7f138c9eddcb0ab4bc4b2b1b6fc51a886eb2aeed11419dbaba9707ad75" gracePeriod=120
Oct 30 16:48:19.036882 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:19.036816    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="66f44d22-9ae0-44aa-a585-cf1370bdb459" containerName="kube-rbac-proxy" containerID="cri-o://f259c4a4af60b21973bcc5de4c39a2b7a7e53616d2af249a91655d87f4594a3c" gracePeriod=120
Oct 30 16:48:19.036882 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:19.036869    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="66f44d22-9ae0-44aa-a585-cf1370bdb459" containerName="prom-label-proxy" containerID="cri-o://51dc261cc66033589c0d98b13207212d0731d2d4767e87a78d2555057a542bb9" gracePeriod=120
Oct 30 16:48:19.037108 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:19.036893    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="66f44d22-9ae0-44aa-a585-cf1370bdb459" containerName="kube-rbac-proxy-web" containerID="cri-o://00d377b51fed238ae4e97d306111b7238f08fd4410c737d3a45aad8dd2b01a94" gracePeriod=120
Oct 30 16:48:19.037108 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:19.036945    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="66f44d22-9ae0-44aa-a585-cf1370bdb459" containerName="config-reloader" containerID="cri-o://c3e0e529d46e7aa7e45bf451750259a576157961cfb41c02df2256229cf388a4" gracePeriod=120
Oct 30 16:48:20.137110 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.137082    4433 generic.go:356] "Generic (PLEG): container finished" podID="66f44d22-9ae0-44aa-a585-cf1370bdb459" containerID="51dc261cc66033589c0d98b13207212d0731d2d4767e87a78d2555057a542bb9" exitCode=0
Oct 30 16:48:20.137110 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.137107    4433 generic.go:356] "Generic (PLEG): container finished" podID="66f44d22-9ae0-44aa-a585-cf1370bdb459" containerID="481f1c7f138c9eddcb0ab4bc4b2b1b6fc51a886eb2aeed11419dbaba9707ad75" exitCode=0
Oct 30 16:48:20.137479 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.137116    4433 generic.go:356] "Generic (PLEG): container finished" podID="66f44d22-9ae0-44aa-a585-cf1370bdb459" containerID="f259c4a4af60b21973bcc5de4c39a2b7a7e53616d2af249a91655d87f4594a3c" exitCode=0
Oct 30 16:48:20.137479 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.137123    4433 generic.go:356] "Generic (PLEG): container finished" podID="66f44d22-9ae0-44aa-a585-cf1370bdb459" containerID="c3e0e529d46e7aa7e45bf451750259a576157961cfb41c02df2256229cf388a4" exitCode=0
Oct 30 16:48:20.137479 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.137129    4433 generic.go:356] "Generic (PLEG): container finished" podID="66f44d22-9ae0-44aa-a585-cf1370bdb459" containerID="c2ca6f96075b549ce738651953f86e0c606fe3ee92a81be204269b727bec787a" exitCode=0
Oct 30 16:48:20.137479 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.137157    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66f44d22-9ae0-44aa-a585-cf1370bdb459","Type":"ContainerDied","Data":"51dc261cc66033589c0d98b13207212d0731d2d4767e87a78d2555057a542bb9"}
Oct 30 16:48:20.137479 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.137228    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66f44d22-9ae0-44aa-a585-cf1370bdb459","Type":"ContainerDied","Data":"481f1c7f138c9eddcb0ab4bc4b2b1b6fc51a886eb2aeed11419dbaba9707ad75"}
Oct 30 16:48:20.137479 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.137254    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66f44d22-9ae0-44aa-a585-cf1370bdb459","Type":"ContainerDied","Data":"f259c4a4af60b21973bcc5de4c39a2b7a7e53616d2af249a91655d87f4594a3c"}
Oct 30 16:48:20.137479 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.137277    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66f44d22-9ae0-44aa-a585-cf1370bdb459","Type":"ContainerDied","Data":"c3e0e529d46e7aa7e45bf451750259a576157961cfb41c02df2256229cf388a4"}
Oct 30 16:48:20.137479 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.137300    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66f44d22-9ae0-44aa-a585-cf1370bdb459","Type":"ContainerDied","Data":"c2ca6f96075b549ce738651953f86e0c606fe3ee92a81be204269b727bec787a"}
Oct 30 16:48:20.306569 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.306531    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:48:20.396717 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.396632    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66f44d22-9ae0-44aa-a585-cf1370bdb459-kube-api-access-45rz9\" (UniqueName: \"kubernetes.io/projected/66f44d22-9ae0-44aa-a585-cf1370bdb459-kube-api-access-45rz9\") pod \"66f44d22-9ae0-44aa-a585-cf1370bdb459\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") "
Oct 30 16:48:20.396717 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.396703    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-secret-alertmanager-main-tls\") pod \"66f44d22-9ae0-44aa-a585-cf1370bdb459\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") "
Oct 30 16:48:20.396925 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.396752    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-cluster-tls-config\") pod \"66f44d22-9ae0-44aa-a585-cf1370bdb459\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") "
Oct 30 16:48:20.396925 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.396804    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-web-config\" (UniqueName: \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-web-config\") pod \"66f44d22-9ae0-44aa-a585-cf1370bdb459\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") "
Oct 30 16:48:20.396925 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.396850    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-config-volume\" (UniqueName: \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-config-volume\") pod \"66f44d22-9ae0-44aa-a585-cf1370bdb459\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") "
Oct 30 16:48:20.396925 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.396884    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66f44d22-9ae0-44aa-a585-cf1370bdb459-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66f44d22-9ae0-44aa-a585-cf1370bdb459-metrics-client-ca\") pod \"66f44d22-9ae0-44aa-a585-cf1370bdb459\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") "
Oct 30 16:48:20.396925 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.396915    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66f44d22-9ae0-44aa-a585-cf1370bdb459-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66f44d22-9ae0-44aa-a585-cf1370bdb459-alertmanager-trusted-ca-bundle\") pod \"66f44d22-9ae0-44aa-a585-cf1370bdb459\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") "
Oct 30 16:48:20.397168 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.396968    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66f44d22-9ae0-44aa-a585-cf1370bdb459-tls-assets\" (UniqueName: \"kubernetes.io/projected/66f44d22-9ae0-44aa-a585-cf1370bdb459-tls-assets\") pod \"66f44d22-9ae0-44aa-a585-cf1370bdb459\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") "
Oct 30 16:48:20.397168 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.397011    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66f44d22-9ae0-44aa-a585-cf1370bdb459-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/66f44d22-9ae0-44aa-a585-cf1370bdb459-alertmanager-main-db\") pod \"66f44d22-9ae0-44aa-a585-cf1370bdb459\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") "
Oct 30 16:48:20.397168 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.397069    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-secret-alertmanager-kube-rbac-proxy\") pod \"66f44d22-9ae0-44aa-a585-cf1370bdb459\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") "
Oct 30 16:48:20.397168 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.397099    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-secret-alertmanager-kube-rbac-proxy-metric\") pod \"66f44d22-9ae0-44aa-a585-cf1370bdb459\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") "
Oct 30 16:48:20.397168 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.397131    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-secret-alertmanager-kube-rbac-proxy-web\") pod \"66f44d22-9ae0-44aa-a585-cf1370bdb459\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") "
Oct 30 16:48:20.397168 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.397161    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66f44d22-9ae0-44aa-a585-cf1370bdb459-config-out\" (UniqueName: \"kubernetes.io/empty-dir/66f44d22-9ae0-44aa-a585-cf1370bdb459-config-out\") pod \"66f44d22-9ae0-44aa-a585-cf1370bdb459\" (UID: \"66f44d22-9ae0-44aa-a585-cf1370bdb459\") "
Oct 30 16:48:20.398290 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.397920    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66f44d22-9ae0-44aa-a585-cf1370bdb459-alertmanager-trusted-ca-bundle" pod "66f44d22-9ae0-44aa-a585-cf1370bdb459" (UID: "66f44d22-9ae0-44aa-a585-cf1370bdb459"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:48:20.398290 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.398114    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66f44d22-9ae0-44aa-a585-cf1370bdb459-metrics-client-ca" pod "66f44d22-9ae0-44aa-a585-cf1370bdb459" (UID: "66f44d22-9ae0-44aa-a585-cf1370bdb459"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:48:20.399991 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.399953    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f44d22-9ae0-44aa-a585-cf1370bdb459-alertmanager-main-db" pod "66f44d22-9ae0-44aa-a585-cf1370bdb459" (UID: "66f44d22-9ae0-44aa-a585-cf1370bdb459"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:48:20.400623 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.400585    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f44d22-9ae0-44aa-a585-cf1370bdb459-config-out" pod "66f44d22-9ae0-44aa-a585-cf1370bdb459" (UID: "66f44d22-9ae0-44aa-a585-cf1370bdb459"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:48:20.406432 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.403852    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66f44d22-9ae0-44aa-a585-cf1370bdb459-kube-api-access-45rz9" pod "66f44d22-9ae0-44aa-a585-cf1370bdb459" (UID: "66f44d22-9ae0-44aa-a585-cf1370bdb459"). InnerVolumeSpecName "kube-api-access-45rz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:48:20.406432 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.403867    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-secret-alertmanager-main-tls" pod "66f44d22-9ae0-44aa-a585-cf1370bdb459" (UID: "66f44d22-9ae0-44aa-a585-cf1370bdb459"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:48:20.406432 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.403987    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-config-volume" pod "66f44d22-9ae0-44aa-a585-cf1370bdb459" (UID: "66f44d22-9ae0-44aa-a585-cf1370bdb459"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:48:20.406432 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.405784    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-secret-alertmanager-kube-rbac-proxy" pod "66f44d22-9ae0-44aa-a585-cf1370bdb459" (UID: "66f44d22-9ae0-44aa-a585-cf1370bdb459"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:48:20.406432 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.406128    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-secret-alertmanager-kube-rbac-proxy-web" pod "66f44d22-9ae0-44aa-a585-cf1370bdb459" (UID: "66f44d22-9ae0-44aa-a585-cf1370bdb459"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:48:20.406761 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.406499    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-secret-alertmanager-kube-rbac-proxy-metric" pod "66f44d22-9ae0-44aa-a585-cf1370bdb459" (UID: "66f44d22-9ae0-44aa-a585-cf1370bdb459"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:48:20.409572 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.409544    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66f44d22-9ae0-44aa-a585-cf1370bdb459-tls-assets" pod "66f44d22-9ae0-44aa-a585-cf1370bdb459" (UID: "66f44d22-9ae0-44aa-a585-cf1370bdb459"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:48:20.410540 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.410515    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-cluster-tls-config" pod "66f44d22-9ae0-44aa-a585-cf1370bdb459" (UID: "66f44d22-9ae0-44aa-a585-cf1370bdb459"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:48:20.417453 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.417428    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-web-config" pod "66f44d22-9ae0-44aa-a585-cf1370bdb459" (UID: "66f44d22-9ae0-44aa-a585-cf1370bdb459"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:48:20.498493 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.498455    4433 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-config-volume\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:20.498493 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.498487    4433 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66f44d22-9ae0-44aa-a585-cf1370bdb459-metrics-client-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:20.498493 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.498499    4433 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66f44d22-9ae0-44aa-a585-cf1370bdb459-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:20.498730 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.498509    4433 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66f44d22-9ae0-44aa-a585-cf1370bdb459-tls-assets\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:20.498730 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.498518    4433 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/66f44d22-9ae0-44aa-a585-cf1370bdb459-alertmanager-main-db\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:20.498730 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.498527    4433 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:20.498730 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.498538    4433 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:20.498730 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.498547    4433 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:20.498730 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.498557    4433 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66f44d22-9ae0-44aa-a585-cf1370bdb459-config-out\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:20.498730 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.498565    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45rz9\" (UniqueName: \"kubernetes.io/projected/66f44d22-9ae0-44aa-a585-cf1370bdb459-kube-api-access-45rz9\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:20.498730 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.498574    4433 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-secret-alertmanager-main-tls\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:20.498730 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.498582    4433 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-cluster-tls-config\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:20.498730 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:20.498611    4433 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66f44d22-9ae0-44aa-a585-cf1370bdb459-web-config\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:21.144105 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:21.144074    4433 generic.go:356] "Generic (PLEG): container finished" podID="66f44d22-9ae0-44aa-a585-cf1370bdb459" containerID="00d377b51fed238ae4e97d306111b7238f08fd4410c737d3a45aad8dd2b01a94" exitCode=0
Oct 30 16:48:21.144496 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:21.144146    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66f44d22-9ae0-44aa-a585-cf1370bdb459","Type":"ContainerDied","Data":"00d377b51fed238ae4e97d306111b7238f08fd4410c737d3a45aad8dd2b01a94"}
Oct 30 16:48:21.144496 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:21.144167    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:48:21.144496 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:21.144207    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66f44d22-9ae0-44aa-a585-cf1370bdb459","Type":"ContainerDied","Data":"dbd4c657a6c968507fafd6096d62c9ad911bc72c8ba5919992700c5ac5934e46"}
Oct 30 16:48:21.144496 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:21.144240    4433 scope.go:122] "RemoveContainer" containerID="51dc261cc66033589c0d98b13207212d0731d2d4767e87a78d2555057a542bb9"
Oct 30 16:48:21.151955 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:21.151881    4433 scope.go:122] "RemoveContainer" containerID="481f1c7f138c9eddcb0ab4bc4b2b1b6fc51a886eb2aeed11419dbaba9707ad75"
Oct 30 16:48:21.159011 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:21.158986    4433 scope.go:122] "RemoveContainer" containerID="f259c4a4af60b21973bcc5de4c39a2b7a7e53616d2af249a91655d87f4594a3c"
Oct 30 16:48:21.166927 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:21.166799    4433 scope.go:122] "RemoveContainer" containerID="00d377b51fed238ae4e97d306111b7238f08fd4410c737d3a45aad8dd2b01a94"
Oct 30 16:48:21.173225 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:21.173207    4433 scope.go:122] "RemoveContainer" containerID="c3e0e529d46e7aa7e45bf451750259a576157961cfb41c02df2256229cf388a4"
Oct 30 16:48:21.179716 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:21.179698    4433 scope.go:122] "RemoveContainer" containerID="c2ca6f96075b549ce738651953f86e0c606fe3ee92a81be204269b727bec787a"
Oct 30 16:48:21.186451 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:21.186435    4433 scope.go:122] "RemoveContainer" containerID="5fa2b6cc88ff8efaf67af4ff267914139bc2ad2ce9904a20c89167c64ee70a55"
Oct 30 16:48:21.193138 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:21.193119    4433 scope.go:122] "RemoveContainer" containerID="51dc261cc66033589c0d98b13207212d0731d2d4767e87a78d2555057a542bb9"
Oct 30 16:48:21.193390 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:48:21.193371    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51dc261cc66033589c0d98b13207212d0731d2d4767e87a78d2555057a542bb9\": container with ID starting with 51dc261cc66033589c0d98b13207212d0731d2d4767e87a78d2555057a542bb9 not found: ID does not exist" containerID="51dc261cc66033589c0d98b13207212d0731d2d4767e87a78d2555057a542bb9"
Oct 30 16:48:21.193468 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:21.193436    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51dc261cc66033589c0d98b13207212d0731d2d4767e87a78d2555057a542bb9"} err="failed to get container status \"51dc261cc66033589c0d98b13207212d0731d2d4767e87a78d2555057a542bb9\": rpc error: code = NotFound desc = could not find container \"51dc261cc66033589c0d98b13207212d0731d2d4767e87a78d2555057a542bb9\": container with ID starting with 51dc261cc66033589c0d98b13207212d0731d2d4767e87a78d2555057a542bb9 not found: ID does not exist"
Oct 30 16:48:21.193542 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:21.193467    4433 scope.go:122] "RemoveContainer" containerID="481f1c7f138c9eddcb0ab4bc4b2b1b6fc51a886eb2aeed11419dbaba9707ad75"
Oct 30 16:48:21.193735 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:48:21.193720    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"481f1c7f138c9eddcb0ab4bc4b2b1b6fc51a886eb2aeed11419dbaba9707ad75\": container with ID starting with 481f1c7f138c9eddcb0ab4bc4b2b1b6fc51a886eb2aeed11419dbaba9707ad75 not found: ID does not exist" containerID="481f1c7f138c9eddcb0ab4bc4b2b1b6fc51a886eb2aeed11419dbaba9707ad75"
Oct 30 16:48:21.193801 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:21.193743    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"481f1c7f138c9eddcb0ab4bc4b2b1b6fc51a886eb2aeed11419dbaba9707ad75"} err="failed to get container status \"481f1c7f138c9eddcb0ab4bc4b2b1b6fc51a886eb2aeed11419dbaba9707ad75\": rpc error: code = NotFound desc = could not find container \"481f1c7f138c9eddcb0ab4bc4b2b1b6fc51a886eb2aeed11419dbaba9707ad75\": container with ID starting with 481f1c7f138c9eddcb0ab4bc4b2b1b6fc51a886eb2aeed11419dbaba9707ad75 not found: ID does not exist"
Oct 30 16:48:21.193801 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:21.193762    4433 scope.go:122] "RemoveContainer" containerID="f259c4a4af60b21973bcc5de4c39a2b7a7e53616d2af249a91655d87f4594a3c"
Oct 30 16:48:21.193997 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:48:21.193977    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f259c4a4af60b21973bcc5de4c39a2b7a7e53616d2af249a91655d87f4594a3c\": container with ID starting with f259c4a4af60b21973bcc5de4c39a2b7a7e53616d2af249a91655d87f4594a3c not found: ID does not exist" containerID="f259c4a4af60b21973bcc5de4c39a2b7a7e53616d2af249a91655d87f4594a3c"
Oct 30 16:48:21.194033 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:21.194005    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f259c4a4af60b21973bcc5de4c39a2b7a7e53616d2af249a91655d87f4594a3c"} err="failed to get container status \"f259c4a4af60b21973bcc5de4c39a2b7a7e53616d2af249a91655d87f4594a3c\": rpc error: code = NotFound desc = could not find container \"f259c4a4af60b21973bcc5de4c39a2b7a7e53616d2af249a91655d87f4594a3c\": container with ID starting with f259c4a4af60b21973bcc5de4c39a2b7a7e53616d2af249a91655d87f4594a3c not found: ID does not exist"
Oct 30 16:48:21.194033 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:21.194020    4433 scope.go:122] "RemoveContainer" containerID="00d377b51fed238ae4e97d306111b7238f08fd4410c737d3a45aad8dd2b01a94"
Oct 30 16:48:21.194230 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:48:21.194213    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00d377b51fed238ae4e97d306111b7238f08fd4410c737d3a45aad8dd2b01a94\": container with ID starting with 00d377b51fed238ae4e97d306111b7238f08fd4410c737d3a45aad8dd2b01a94 not found: ID does not exist" containerID="00d377b51fed238ae4e97d306111b7238f08fd4410c737d3a45aad8dd2b01a94"
Oct 30 16:48:21.194267 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:21.194236    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00d377b51fed238ae4e97d306111b7238f08fd4410c737d3a45aad8dd2b01a94"} err="failed to get container status \"00d377b51fed238ae4e97d306111b7238f08fd4410c737d3a45aad8dd2b01a94\": rpc error: code = NotFound desc = could not find container \"00d377b51fed238ae4e97d306111b7238f08fd4410c737d3a45aad8dd2b01a94\": container with ID starting with 00d377b51fed238ae4e97d306111b7238f08fd4410c737d3a45aad8dd2b01a94 not found: ID does not exist"
Oct 30 16:48:21.194267 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:21.194251    4433 scope.go:122] "RemoveContainer" containerID="c3e0e529d46e7aa7e45bf451750259a576157961cfb41c02df2256229cf388a4"
Oct 30 16:48:21.194471 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:48:21.194457    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3e0e529d46e7aa7e45bf451750259a576157961cfb41c02df2256229cf388a4\": container with ID starting with c3e0e529d46e7aa7e45bf451750259a576157961cfb41c02df2256229cf388a4 not found: ID does not exist" containerID="c3e0e529d46e7aa7e45bf451750259a576157961cfb41c02df2256229cf388a4"
Oct 30 16:48:21.194510 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:21.194473    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3e0e529d46e7aa7e45bf451750259a576157961cfb41c02df2256229cf388a4"} err="failed to get container status \"c3e0e529d46e7aa7e45bf451750259a576157961cfb41c02df2256229cf388a4\": rpc error: code = NotFound desc = could not find container \"c3e0e529d46e7aa7e45bf451750259a576157961cfb41c02df2256229cf388a4\": container with ID starting with c3e0e529d46e7aa7e45bf451750259a576157961cfb41c02df2256229cf388a4 not found: ID does not exist"
Oct 30 16:48:21.194510 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:21.194488    4433 scope.go:122] "RemoveContainer" containerID="c2ca6f96075b549ce738651953f86e0c606fe3ee92a81be204269b727bec787a"
Oct 30 16:48:21.194695 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:48:21.194678    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2ca6f96075b549ce738651953f86e0c606fe3ee92a81be204269b727bec787a\": container with ID starting with c2ca6f96075b549ce738651953f86e0c606fe3ee92a81be204269b727bec787a not found: ID does not exist" containerID="c2ca6f96075b549ce738651953f86e0c606fe3ee92a81be204269b727bec787a"
Oct 30 16:48:21.194754 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:21.194703    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2ca6f96075b549ce738651953f86e0c606fe3ee92a81be204269b727bec787a"} err="failed to get container status \"c2ca6f96075b549ce738651953f86e0c606fe3ee92a81be204269b727bec787a\": rpc error: code = NotFound desc = could not find container \"c2ca6f96075b549ce738651953f86e0c606fe3ee92a81be204269b727bec787a\": container with ID starting with c2ca6f96075b549ce738651953f86e0c606fe3ee92a81be204269b727bec787a not found: ID does not exist"
Oct 30 16:48:21.194754 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:21.194722    4433 scope.go:122] "RemoveContainer" containerID="5fa2b6cc88ff8efaf67af4ff267914139bc2ad2ce9904a20c89167c64ee70a55"
Oct 30 16:48:21.194932 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:48:21.194917    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fa2b6cc88ff8efaf67af4ff267914139bc2ad2ce9904a20c89167c64ee70a55\": container with ID starting with 5fa2b6cc88ff8efaf67af4ff267914139bc2ad2ce9904a20c89167c64ee70a55 not found: ID does not exist" containerID="5fa2b6cc88ff8efaf67af4ff267914139bc2ad2ce9904a20c89167c64ee70a55"
Oct 30 16:48:21.194971 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:21.194934    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fa2b6cc88ff8efaf67af4ff267914139bc2ad2ce9904a20c89167c64ee70a55"} err="failed to get container status \"5fa2b6cc88ff8efaf67af4ff267914139bc2ad2ce9904a20c89167c64ee70a55\": rpc error: code = NotFound desc = could not find container \"5fa2b6cc88ff8efaf67af4ff267914139bc2ad2ce9904a20c89167c64ee70a55\": container with ID starting with 5fa2b6cc88ff8efaf67af4ff267914139bc2ad2ce9904a20c89167c64ee70a55 not found: ID does not exist"
Oct 30 16:48:21.208358 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:21.208337    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 16:48:21.220013 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:21.219994    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 16:48:21.693266 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:21.693223    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66f44d22-9ae0-44aa-a585-cf1370bdb459" path="/var/lib/kubelet/pods/66f44d22-9ae0-44aa-a585-cf1370bdb459/volumes"
Oct 30 16:48:23.666955 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:23.666919    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 16:48:23.667492 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:23.667288    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0dcced64-e20a-4247-aa7d-314a1e2de54a" containerName="prometheus" containerID="cri-o://09297173c46029a0dca8195cac8af92f1dcc3d1c1fb8490d75361abe48921af3" gracePeriod=600
Oct 30 16:48:23.667740 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:23.667710    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0dcced64-e20a-4247-aa7d-314a1e2de54a" containerName="kube-rbac-proxy-thanos" containerID="cri-o://35e77c7bf4dd7af16a9cd4fd165e0c61e2920ca2efbd1b51d08f3c7faefaa416" gracePeriod=600
Oct 30 16:48:23.669665 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:23.668988    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0dcced64-e20a-4247-aa7d-314a1e2de54a" containerName="thanos-sidecar" containerID="cri-o://1f629cbf1c6bc48868eff9832329db0ba715fb878d2ea1b62b4b1f49448df1e3" gracePeriod=600
Oct 30 16:48:23.670491 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:23.669064    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0dcced64-e20a-4247-aa7d-314a1e2de54a" containerName="kube-rbac-proxy" containerID="cri-o://7b75b3baa9832ef9cae5c4a477577449f821a3fa40e67f12a1c641a22096e95c" gracePeriod=600
Oct 30 16:48:23.670600 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:23.669083    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0dcced64-e20a-4247-aa7d-314a1e2de54a" containerName="kube-rbac-proxy-web" containerID="cri-o://114dcb058dd2d13daf31e393642b8b3b66d0c45c62ddf482a476c9e71c7b8ca1" gracePeriod=600
Oct 30 16:48:23.670659 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:23.669121    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0dcced64-e20a-4247-aa7d-314a1e2de54a" containerName="config-reloader" containerID="cri-o://d482751a418acea4811f69cfe03140b0c77049c0db4a4c8f59c2f8e0acecad78" gracePeriod=600
Oct 30 16:48:24.164521 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:24.164486    4433 generic.go:356] "Generic (PLEG): container finished" podID="0dcced64-e20a-4247-aa7d-314a1e2de54a" containerID="35e77c7bf4dd7af16a9cd4fd165e0c61e2920ca2efbd1b51d08f3c7faefaa416" exitCode=0
Oct 30 16:48:24.164521 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:24.164511    4433 generic.go:356] "Generic (PLEG): container finished" podID="0dcced64-e20a-4247-aa7d-314a1e2de54a" containerID="7b75b3baa9832ef9cae5c4a477577449f821a3fa40e67f12a1c641a22096e95c" exitCode=0
Oct 30 16:48:24.164521 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:24.164518    4433 generic.go:356] "Generic (PLEG): container finished" podID="0dcced64-e20a-4247-aa7d-314a1e2de54a" containerID="1f629cbf1c6bc48868eff9832329db0ba715fb878d2ea1b62b4b1f49448df1e3" exitCode=0
Oct 30 16:48:24.164521 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:24.164524    4433 generic.go:356] "Generic (PLEG): container finished" podID="0dcced64-e20a-4247-aa7d-314a1e2de54a" containerID="d482751a418acea4811f69cfe03140b0c77049c0db4a4c8f59c2f8e0acecad78" exitCode=0
Oct 30 16:48:24.164521 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:24.164529    4433 generic.go:356] "Generic (PLEG): container finished" podID="0dcced64-e20a-4247-aa7d-314a1e2de54a" containerID="09297173c46029a0dca8195cac8af92f1dcc3d1c1fb8490d75361abe48921af3" exitCode=0
Oct 30 16:48:24.164876 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:24.164559    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0dcced64-e20a-4247-aa7d-314a1e2de54a","Type":"ContainerDied","Data":"35e77c7bf4dd7af16a9cd4fd165e0c61e2920ca2efbd1b51d08f3c7faefaa416"}
Oct 30 16:48:24.164876 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:24.164647    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0dcced64-e20a-4247-aa7d-314a1e2de54a","Type":"ContainerDied","Data":"7b75b3baa9832ef9cae5c4a477577449f821a3fa40e67f12a1c641a22096e95c"}
Oct 30 16:48:24.164876 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:24.164685    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0dcced64-e20a-4247-aa7d-314a1e2de54a","Type":"ContainerDied","Data":"1f629cbf1c6bc48868eff9832329db0ba715fb878d2ea1b62b4b1f49448df1e3"}
Oct 30 16:48:24.164876 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:24.164726    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0dcced64-e20a-4247-aa7d-314a1e2de54a","Type":"ContainerDied","Data":"d482751a418acea4811f69cfe03140b0c77049c0db4a4c8f59c2f8e0acecad78"}
Oct 30 16:48:24.164876 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:24.164762    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0dcced64-e20a-4247-aa7d-314a1e2de54a","Type":"ContainerDied","Data":"09297173c46029a0dca8195cac8af92f1dcc3d1c1fb8490d75361abe48921af3"}
Oct 30 16:48:24.934975 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:24.934949    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.038108 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.038033    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-configmap-metrics-client-ca\") pod \"0dcced64-e20a-4247-aa7d-314a1e2de54a\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") "
Oct 30 16:48:25.038108 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.038087    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-prometheus-k8s-rulefiles-2\") pod \"0dcced64-e20a-4247-aa7d-314a1e2de54a\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") "
Oct 30 16:48:25.038329 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.038110    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-metrics-client-certs\") pod \"0dcced64-e20a-4247-aa7d-314a1e2de54a\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") "
Oct 30 16:48:25.038329 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.038130    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-prometheus-k8s-tls\") pod \"0dcced64-e20a-4247-aa7d-314a1e2de54a\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") "
Oct 30 16:48:25.038329 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.038166    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-kube-rbac-proxy\") pod \"0dcced64-e20a-4247-aa7d-314a1e2de54a\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") "
Oct 30 16:48:25.038329 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.038192    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"0dcced64-e20a-4247-aa7d-314a1e2de54a\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") "
Oct 30 16:48:25.038329 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.038226    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-thanos-prometheus-http-client-file\") pod \"0dcced64-e20a-4247-aa7d-314a1e2de54a\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") "
Oct 30 16:48:25.038329 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.038258    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-configmap-serving-certs-ca-bundle\") pod \"0dcced64-e20a-4247-aa7d-314a1e2de54a\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") "
Oct 30 16:48:25.038329 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.038288    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-prometheus-k8s-rulefiles-0\") pod \"0dcced64-e20a-4247-aa7d-314a1e2de54a\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") "
Oct 30 16:48:25.038728 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.038334    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dcced64-e20a-4247-aa7d-314a1e2de54a-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0dcced64-e20a-4247-aa7d-314a1e2de54a-prometheus-k8s-db\") pod \"0dcced64-e20a-4247-aa7d-314a1e2de54a\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") "
Oct 30 16:48:25.038728 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.038374    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-configmap-kubelet-serving-ca-bundle\") pod \"0dcced64-e20a-4247-aa7d-314a1e2de54a\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") "
Oct 30 16:48:25.038728 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.038443    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dcced64-e20a-4247-aa7d-314a1e2de54a-tls-assets\" (UniqueName: \"kubernetes.io/projected/0dcced64-e20a-4247-aa7d-314a1e2de54a-tls-assets\") pod \"0dcced64-e20a-4247-aa7d-314a1e2de54a\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") "
Oct 30 16:48:25.038728 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.038473    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-prometheus-trusted-ca-bundle\") pod \"0dcced64-e20a-4247-aa7d-314a1e2de54a\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") "
Oct 30 16:48:25.038728 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.038502    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-grpc-tls\") pod \"0dcced64-e20a-4247-aa7d-314a1e2de54a\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") "
Oct 30 16:48:25.038728 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.038504    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-configmap-metrics-client-ca" pod "0dcced64-e20a-4247-aa7d-314a1e2de54a" (UID: "0dcced64-e20a-4247-aa7d-314a1e2de54a"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:48:25.038728 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.038537    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dcced64-e20a-4247-aa7d-314a1e2de54a-kube-api-access-j8pln\" (UniqueName: \"kubernetes.io/projected/0dcced64-e20a-4247-aa7d-314a1e2de54a-kube-api-access-j8pln\") pod \"0dcced64-e20a-4247-aa7d-314a1e2de54a\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") "
Oct 30 16:48:25.038728 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.038575    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dcced64-e20a-4247-aa7d-314a1e2de54a-config-out\" (UniqueName: \"kubernetes.io/empty-dir/0dcced64-e20a-4247-aa7d-314a1e2de54a-config-out\") pod \"0dcced64-e20a-4247-aa7d-314a1e2de54a\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") "
Oct 30 16:48:25.038728 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.038638    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-config\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-config\") pod \"0dcced64-e20a-4247-aa7d-314a1e2de54a\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") "
Oct 30 16:48:25.038728 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.038598    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-prometheus-k8s-rulefiles-2" pod "0dcced64-e20a-4247-aa7d-314a1e2de54a" (UID: "0dcced64-e20a-4247-aa7d-314a1e2de54a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:48:25.038728 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.038682    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-web-config\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-web-config\") pod \"0dcced64-e20a-4247-aa7d-314a1e2de54a\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") "
Oct 30 16:48:25.039271 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.038943    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-configmap-kubelet-serving-ca-bundle" pod "0dcced64-e20a-4247-aa7d-314a1e2de54a" (UID: "0dcced64-e20a-4247-aa7d-314a1e2de54a"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:48:25.039785 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.039508    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-configmap-serving-certs-ca-bundle" pod "0dcced64-e20a-4247-aa7d-314a1e2de54a" (UID: "0dcced64-e20a-4247-aa7d-314a1e2de54a"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:48:25.039971 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.039943    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-prometheus-k8s-rulefiles-0" pod "0dcced64-e20a-4247-aa7d-314a1e2de54a" (UID: "0dcced64-e20a-4247-aa7d-314a1e2de54a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:48:25.040072 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.040001    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-prometheus-k8s-rulefiles-1\") pod \"0dcced64-e20a-4247-aa7d-314a1e2de54a\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") "
Oct 30 16:48:25.040072 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.040039    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"0dcced64-e20a-4247-aa7d-314a1e2de54a\" (UID: \"0dcced64-e20a-4247-aa7d-314a1e2de54a\") "
Oct 30 16:48:25.040278 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.040255    4433 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:25.040431 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.040280    4433 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-configmap-metrics-client-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:25.040431 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.040295    4433 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:25.040431 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.040308    4433 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:25.040431 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.040320    4433 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:25.043029 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.042956    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dcced64-e20a-4247-aa7d-314a1e2de54a-prometheus-k8s-db" pod "0dcced64-e20a-4247-aa7d-314a1e2de54a" (UID: "0dcced64-e20a-4247-aa7d-314a1e2de54a"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:48:25.043340 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.043312    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dcced64-e20a-4247-aa7d-314a1e2de54a-kube-api-access-j8pln" pod "0dcced64-e20a-4247-aa7d-314a1e2de54a" (UID: "0dcced64-e20a-4247-aa7d-314a1e2de54a"). InnerVolumeSpecName "kube-api-access-j8pln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:48:25.043454 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.043347    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-prometheus-k8s-rulefiles-1" pod "0dcced64-e20a-4247-aa7d-314a1e2de54a" (UID: "0dcced64-e20a-4247-aa7d-314a1e2de54a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:48:25.043626 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.043594    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-prometheus-k8s-thanos-sidecar-tls" pod "0dcced64-e20a-4247-aa7d-314a1e2de54a" (UID: "0dcced64-e20a-4247-aa7d-314a1e2de54a"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:48:25.043721 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.043702    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-prometheus-trusted-ca-bundle" pod "0dcced64-e20a-4247-aa7d-314a1e2de54a" (UID: "0dcced64-e20a-4247-aa7d-314a1e2de54a"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:48:25.043797 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.043773    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-thanos-prometheus-http-client-file" pod "0dcced64-e20a-4247-aa7d-314a1e2de54a" (UID: "0dcced64-e20a-4247-aa7d-314a1e2de54a"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:48:25.045846 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.045818    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dcced64-e20a-4247-aa7d-314a1e2de54a-config-out" pod "0dcced64-e20a-4247-aa7d-314a1e2de54a" (UID: "0dcced64-e20a-4247-aa7d-314a1e2de54a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:48:25.046053 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.046029    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-prometheus-k8s-tls" pod "0dcced64-e20a-4247-aa7d-314a1e2de54a" (UID: "0dcced64-e20a-4247-aa7d-314a1e2de54a"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:48:25.046148 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.046130    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-metrics-client-certs" pod "0dcced64-e20a-4247-aa7d-314a1e2de54a" (UID: "0dcced64-e20a-4247-aa7d-314a1e2de54a"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:48:25.046200 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.046158    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-grpc-tls" pod "0dcced64-e20a-4247-aa7d-314a1e2de54a" (UID: "0dcced64-e20a-4247-aa7d-314a1e2de54a"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:48:25.046443 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.046394    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-config" pod "0dcced64-e20a-4247-aa7d-314a1e2de54a" (UID: "0dcced64-e20a-4247-aa7d-314a1e2de54a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:48:25.046612 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.046592    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-kube-rbac-proxy" pod "0dcced64-e20a-4247-aa7d-314a1e2de54a" (UID: "0dcced64-e20a-4247-aa7d-314a1e2de54a"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:48:25.047488 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.047455    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-prometheus-k8s-kube-rbac-proxy-web" pod "0dcced64-e20a-4247-aa7d-314a1e2de54a" (UID: "0dcced64-e20a-4247-aa7d-314a1e2de54a"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:48:25.047649 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.047619    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dcced64-e20a-4247-aa7d-314a1e2de54a-tls-assets" pod "0dcced64-e20a-4247-aa7d-314a1e2de54a" (UID: "0dcced64-e20a-4247-aa7d-314a1e2de54a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:48:25.059030 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.058996    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-web-config" pod "0dcced64-e20a-4247-aa7d-314a1e2de54a" (UID: "0dcced64-e20a-4247-aa7d-314a1e2de54a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:48:25.140718 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.140677    4433 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-metrics-client-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:25.140718 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.140713    4433 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-prometheus-k8s-tls\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:25.140718 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.140723    4433 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-kube-rbac-proxy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:25.140971 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.140734    4433 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:25.140971 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.140745    4433 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-thanos-prometheus-http-client-file\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:25.140971 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.140754    4433 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0dcced64-e20a-4247-aa7d-314a1e2de54a-prometheus-k8s-db\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:25.140971 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.140764    4433 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0dcced64-e20a-4247-aa7d-314a1e2de54a-tls-assets\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:25.140971 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.140772    4433 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-prometheus-trusted-ca-bundle\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:25.140971 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.140781    4433 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-grpc-tls\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:25.140971 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.140789    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8pln\" (UniqueName: \"kubernetes.io/projected/0dcced64-e20a-4247-aa7d-314a1e2de54a-kube-api-access-j8pln\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:25.140971 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.140797    4433 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0dcced64-e20a-4247-aa7d-314a1e2de54a-config-out\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:25.140971 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.140806    4433 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-config\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:25.140971 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.140835    4433 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-web-config\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:25.140971 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.140843    4433 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0dcced64-e20a-4247-aa7d-314a1e2de54a-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:25.140971 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.140852    4433 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0dcced64-e20a-4247-aa7d-314a1e2de54a-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:25.170962 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.170932    4433 generic.go:356] "Generic (PLEG): container finished" podID="0dcced64-e20a-4247-aa7d-314a1e2de54a" containerID="114dcb058dd2d13daf31e393642b8b3b66d0c45c62ddf482a476c9e71c7b8ca1" exitCode=0
Oct 30 16:48:25.171125 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.170989    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0dcced64-e20a-4247-aa7d-314a1e2de54a","Type":"ContainerDied","Data":"114dcb058dd2d13daf31e393642b8b3b66d0c45c62ddf482a476c9e71c7b8ca1"}
Oct 30 16:48:25.171125 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.171025    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.171125 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.171046    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0dcced64-e20a-4247-aa7d-314a1e2de54a","Type":"ContainerDied","Data":"010a16365b187fcd2ab284a87d3277005a7cc790f031ca2b3e56e392d93e17a7"}
Oct 30 16:48:25.171125 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.171078    4433 scope.go:122] "RemoveContainer" containerID="35e77c7bf4dd7af16a9cd4fd165e0c61e2920ca2efbd1b51d08f3c7faefaa416"
Oct 30 16:48:25.178663 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.178638    4433 scope.go:122] "RemoveContainer" containerID="7b75b3baa9832ef9cae5c4a477577449f821a3fa40e67f12a1c641a22096e95c"
Oct 30 16:48:25.187798 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.187775    4433 scope.go:122] "RemoveContainer" containerID="114dcb058dd2d13daf31e393642b8b3b66d0c45c62ddf482a476c9e71c7b8ca1"
Oct 30 16:48:25.195091 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.195072    4433 scope.go:122] "RemoveContainer" containerID="1f629cbf1c6bc48868eff9832329db0ba715fb878d2ea1b62b4b1f49448df1e3"
Oct 30 16:48:25.201468 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.201448    4433 scope.go:122] "RemoveContainer" containerID="d482751a418acea4811f69cfe03140b0c77049c0db4a4c8f59c2f8e0acecad78"
Oct 30 16:48:25.208595 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.208567    4433 scope.go:122] "RemoveContainer" containerID="09297173c46029a0dca8195cac8af92f1dcc3d1c1fb8490d75361abe48921af3"
Oct 30 16:48:25.215489 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.215466    4433 scope.go:122] "RemoveContainer" containerID="45fd61c193da51e770f95894b7fcc5cd41594c13f8db715997019199eb8d57ce"
Oct 30 16:48:25.222099 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.222081    4433 scope.go:122] "RemoveContainer" containerID="35e77c7bf4dd7af16a9cd4fd165e0c61e2920ca2efbd1b51d08f3c7faefaa416"
Oct 30 16:48:25.222426 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:48:25.222388    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35e77c7bf4dd7af16a9cd4fd165e0c61e2920ca2efbd1b51d08f3c7faefaa416\": container with ID starting with 35e77c7bf4dd7af16a9cd4fd165e0c61e2920ca2efbd1b51d08f3c7faefaa416 not found: ID does not exist" containerID="35e77c7bf4dd7af16a9cd4fd165e0c61e2920ca2efbd1b51d08f3c7faefaa416"
Oct 30 16:48:25.222504 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.222436    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35e77c7bf4dd7af16a9cd4fd165e0c61e2920ca2efbd1b51d08f3c7faefaa416"} err="failed to get container status \"35e77c7bf4dd7af16a9cd4fd165e0c61e2920ca2efbd1b51d08f3c7faefaa416\": rpc error: code = NotFound desc = could not find container \"35e77c7bf4dd7af16a9cd4fd165e0c61e2920ca2efbd1b51d08f3c7faefaa416\": container with ID starting with 35e77c7bf4dd7af16a9cd4fd165e0c61e2920ca2efbd1b51d08f3c7faefaa416 not found: ID does not exist"
Oct 30 16:48:25.222504 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.222455    4433 scope.go:122] "RemoveContainer" containerID="7b75b3baa9832ef9cae5c4a477577449f821a3fa40e67f12a1c641a22096e95c"
Oct 30 16:48:25.222698 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:48:25.222678    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b75b3baa9832ef9cae5c4a477577449f821a3fa40e67f12a1c641a22096e95c\": container with ID starting with 7b75b3baa9832ef9cae5c4a477577449f821a3fa40e67f12a1c641a22096e95c not found: ID does not exist" containerID="7b75b3baa9832ef9cae5c4a477577449f821a3fa40e67f12a1c641a22096e95c"
Oct 30 16:48:25.222741 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.222706    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b75b3baa9832ef9cae5c4a477577449f821a3fa40e67f12a1c641a22096e95c"} err="failed to get container status \"7b75b3baa9832ef9cae5c4a477577449f821a3fa40e67f12a1c641a22096e95c\": rpc error: code = NotFound desc = could not find container \"7b75b3baa9832ef9cae5c4a477577449f821a3fa40e67f12a1c641a22096e95c\": container with ID starting with 7b75b3baa9832ef9cae5c4a477577449f821a3fa40e67f12a1c641a22096e95c not found: ID does not exist"
Oct 30 16:48:25.222741 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.222722    4433 scope.go:122] "RemoveContainer" containerID="114dcb058dd2d13daf31e393642b8b3b66d0c45c62ddf482a476c9e71c7b8ca1"
Oct 30 16:48:25.222943 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:48:25.222926    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"114dcb058dd2d13daf31e393642b8b3b66d0c45c62ddf482a476c9e71c7b8ca1\": container with ID starting with 114dcb058dd2d13daf31e393642b8b3b66d0c45c62ddf482a476c9e71c7b8ca1 not found: ID does not exist" containerID="114dcb058dd2d13daf31e393642b8b3b66d0c45c62ddf482a476c9e71c7b8ca1"
Oct 30 16:48:25.223004 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.222951    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"114dcb058dd2d13daf31e393642b8b3b66d0c45c62ddf482a476c9e71c7b8ca1"} err="failed to get container status \"114dcb058dd2d13daf31e393642b8b3b66d0c45c62ddf482a476c9e71c7b8ca1\": rpc error: code = NotFound desc = could not find container \"114dcb058dd2d13daf31e393642b8b3b66d0c45c62ddf482a476c9e71c7b8ca1\": container with ID starting with 114dcb058dd2d13daf31e393642b8b3b66d0c45c62ddf482a476c9e71c7b8ca1 not found: ID does not exist"
Oct 30 16:48:25.223004 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.222973    4433 scope.go:122] "RemoveContainer" containerID="1f629cbf1c6bc48868eff9832329db0ba715fb878d2ea1b62b4b1f49448df1e3"
Oct 30 16:48:25.223235 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:48:25.223219    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f629cbf1c6bc48868eff9832329db0ba715fb878d2ea1b62b4b1f49448df1e3\": container with ID starting with 1f629cbf1c6bc48868eff9832329db0ba715fb878d2ea1b62b4b1f49448df1e3 not found: ID does not exist" containerID="1f629cbf1c6bc48868eff9832329db0ba715fb878d2ea1b62b4b1f49448df1e3"
Oct 30 16:48:25.223272 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.223240    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f629cbf1c6bc48868eff9832329db0ba715fb878d2ea1b62b4b1f49448df1e3"} err="failed to get container status \"1f629cbf1c6bc48868eff9832329db0ba715fb878d2ea1b62b4b1f49448df1e3\": rpc error: code = NotFound desc = could not find container \"1f629cbf1c6bc48868eff9832329db0ba715fb878d2ea1b62b4b1f49448df1e3\": container with ID starting with 1f629cbf1c6bc48868eff9832329db0ba715fb878d2ea1b62b4b1f49448df1e3 not found: ID does not exist"
Oct 30 16:48:25.223272 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.223253    4433 scope.go:122] "RemoveContainer" containerID="d482751a418acea4811f69cfe03140b0c77049c0db4a4c8f59c2f8e0acecad78"
Oct 30 16:48:25.223503 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:48:25.223468    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d482751a418acea4811f69cfe03140b0c77049c0db4a4c8f59c2f8e0acecad78\": container with ID starting with d482751a418acea4811f69cfe03140b0c77049c0db4a4c8f59c2f8e0acecad78 not found: ID does not exist" containerID="d482751a418acea4811f69cfe03140b0c77049c0db4a4c8f59c2f8e0acecad78"
Oct 30 16:48:25.223563 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.223510    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d482751a418acea4811f69cfe03140b0c77049c0db4a4c8f59c2f8e0acecad78"} err="failed to get container status \"d482751a418acea4811f69cfe03140b0c77049c0db4a4c8f59c2f8e0acecad78\": rpc error: code = NotFound desc = could not find container \"d482751a418acea4811f69cfe03140b0c77049c0db4a4c8f59c2f8e0acecad78\": container with ID starting with d482751a418acea4811f69cfe03140b0c77049c0db4a4c8f59c2f8e0acecad78 not found: ID does not exist"
Oct 30 16:48:25.223563 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.223526    4433 scope.go:122] "RemoveContainer" containerID="09297173c46029a0dca8195cac8af92f1dcc3d1c1fb8490d75361abe48921af3"
Oct 30 16:48:25.223769 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:48:25.223753    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09297173c46029a0dca8195cac8af92f1dcc3d1c1fb8490d75361abe48921af3\": container with ID starting with 09297173c46029a0dca8195cac8af92f1dcc3d1c1fb8490d75361abe48921af3 not found: ID does not exist" containerID="09297173c46029a0dca8195cac8af92f1dcc3d1c1fb8490d75361abe48921af3"
Oct 30 16:48:25.223829 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.223776    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09297173c46029a0dca8195cac8af92f1dcc3d1c1fb8490d75361abe48921af3"} err="failed to get container status \"09297173c46029a0dca8195cac8af92f1dcc3d1c1fb8490d75361abe48921af3\": rpc error: code = NotFound desc = could not find container \"09297173c46029a0dca8195cac8af92f1dcc3d1c1fb8490d75361abe48921af3\": container with ID starting with 09297173c46029a0dca8195cac8af92f1dcc3d1c1fb8490d75361abe48921af3 not found: ID does not exist"
Oct 30 16:48:25.223829 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.223796    4433 scope.go:122] "RemoveContainer" containerID="45fd61c193da51e770f95894b7fcc5cd41594c13f8db715997019199eb8d57ce"
Oct 30 16:48:25.224034 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:48:25.224017    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45fd61c193da51e770f95894b7fcc5cd41594c13f8db715997019199eb8d57ce\": container with ID starting with 45fd61c193da51e770f95894b7fcc5cd41594c13f8db715997019199eb8d57ce not found: ID does not exist" containerID="45fd61c193da51e770f95894b7fcc5cd41594c13f8db715997019199eb8d57ce"
Oct 30 16:48:25.224085 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.224041    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45fd61c193da51e770f95894b7fcc5cd41594c13f8db715997019199eb8d57ce"} err="failed to get container status \"45fd61c193da51e770f95894b7fcc5cd41594c13f8db715997019199eb8d57ce\": rpc error: code = NotFound desc = could not find container \"45fd61c193da51e770f95894b7fcc5cd41594c13f8db715997019199eb8d57ce\": container with ID starting with 45fd61c193da51e770f95894b7fcc5cd41594c13f8db715997019199eb8d57ce not found: ID does not exist"
Oct 30 16:48:25.226787 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.226766    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 16:48:25.232200 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.232178    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 16:48:25.267247 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267216    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267550    4433 cpu_manager.go:414] "containerMap: removing container" podUID="0dcced64-e20a-4247-aa7d-314a1e2de54a" containerName="init-config-reloader"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267571    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dcced64-e20a-4247-aa7d-314a1e2de54a" containerName="init-config-reloader"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267586    4433 cpu_manager.go:414] "containerMap: removing container" podUID="0dcced64-e20a-4247-aa7d-314a1e2de54a" containerName="thanos-sidecar"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267593    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dcced64-e20a-4247-aa7d-314a1e2de54a" containerName="thanos-sidecar"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267607    4433 cpu_manager.go:414] "containerMap: removing container" podUID="0dcced64-e20a-4247-aa7d-314a1e2de54a" containerName="kube-rbac-proxy-web"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267614    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dcced64-e20a-4247-aa7d-314a1e2de54a" containerName="kube-rbac-proxy-web"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267622    4433 cpu_manager.go:414] "containerMap: removing container" podUID="0dcced64-e20a-4247-aa7d-314a1e2de54a" containerName="kube-rbac-proxy"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267629    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dcced64-e20a-4247-aa7d-314a1e2de54a" containerName="kube-rbac-proxy"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267639    4433 cpu_manager.go:414] "containerMap: removing container" podUID="66f44d22-9ae0-44aa-a585-cf1370bdb459" containerName="alertmanager"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267644    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66f44d22-9ae0-44aa-a585-cf1370bdb459" containerName="alertmanager"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267649    4433 cpu_manager.go:414] "containerMap: removing container" podUID="0dcced64-e20a-4247-aa7d-314a1e2de54a" containerName="prometheus"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267654    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dcced64-e20a-4247-aa7d-314a1e2de54a" containerName="prometheus"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267661    4433 cpu_manager.go:414] "containerMap: removing container" podUID="0dcced64-e20a-4247-aa7d-314a1e2de54a" containerName="kube-rbac-proxy-thanos"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267665    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dcced64-e20a-4247-aa7d-314a1e2de54a" containerName="kube-rbac-proxy-thanos"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267677    4433 cpu_manager.go:414] "containerMap: removing container" podUID="66f44d22-9ae0-44aa-a585-cf1370bdb459" containerName="kube-rbac-proxy-web"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267682    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66f44d22-9ae0-44aa-a585-cf1370bdb459" containerName="kube-rbac-proxy-web"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267692    4433 cpu_manager.go:414] "containerMap: removing container" podUID="0dcced64-e20a-4247-aa7d-314a1e2de54a" containerName="config-reloader"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267697    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dcced64-e20a-4247-aa7d-314a1e2de54a" containerName="config-reloader"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267703    4433 cpu_manager.go:414] "containerMap: removing container" podUID="66f44d22-9ae0-44aa-a585-cf1370bdb459" containerName="kube-rbac-proxy"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267707    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66f44d22-9ae0-44aa-a585-cf1370bdb459" containerName="kube-rbac-proxy"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267713    4433 cpu_manager.go:414] "containerMap: removing container" podUID="66f44d22-9ae0-44aa-a585-cf1370bdb459" containerName="kube-rbac-proxy-metric"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267717    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66f44d22-9ae0-44aa-a585-cf1370bdb459" containerName="kube-rbac-proxy-metric"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267727    4433 cpu_manager.go:414] "containerMap: removing container" podUID="66f44d22-9ae0-44aa-a585-cf1370bdb459" containerName="config-reloader"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267752    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66f44d22-9ae0-44aa-a585-cf1370bdb459" containerName="config-reloader"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267763    4433 cpu_manager.go:414] "containerMap: removing container" podUID="5a9be69d-b840-4887-8a9a-822879938df1" containerName="console"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267768    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a9be69d-b840-4887-8a9a-822879938df1" containerName="console"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267774    4433 cpu_manager.go:414] "containerMap: removing container" podUID="66f44d22-9ae0-44aa-a585-cf1370bdb459" containerName="init-config-reloader"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267779    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66f44d22-9ae0-44aa-a585-cf1370bdb459" containerName="init-config-reloader"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267787    4433 cpu_manager.go:414] "containerMap: removing container" podUID="66f44d22-9ae0-44aa-a585-cf1370bdb459" containerName="prom-label-proxy"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267792    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66f44d22-9ae0-44aa-a585-cf1370bdb459" containerName="prom-label-proxy"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267846    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dcced64-e20a-4247-aa7d-314a1e2de54a" containerName="config-reloader"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267857    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="66f44d22-9ae0-44aa-a585-cf1370bdb459" containerName="kube-rbac-proxy"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267864    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="66f44d22-9ae0-44aa-a585-cf1370bdb459" containerName="kube-rbac-proxy-metric"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267871    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="66f44d22-9ae0-44aa-a585-cf1370bdb459" containerName="config-reloader"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267877    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a9be69d-b840-4887-8a9a-822879938df1" containerName="console"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267883    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="66f44d22-9ae0-44aa-a585-cf1370bdb459" containerName="prom-label-proxy"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267889    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dcced64-e20a-4247-aa7d-314a1e2de54a" containerName="thanos-sidecar"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267895    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dcced64-e20a-4247-aa7d-314a1e2de54a" containerName="kube-rbac-proxy-web"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267899    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dcced64-e20a-4247-aa7d-314a1e2de54a" containerName="kube-rbac-proxy"
Oct 30 16:48:25.267875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267904    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="66f44d22-9ae0-44aa-a585-cf1370bdb459" containerName="alertmanager"
Oct 30 16:48:25.269067 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267910    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dcced64-e20a-4247-aa7d-314a1e2de54a" containerName="prometheus"
Oct 30 16:48:25.269067 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267918    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dcced64-e20a-4247-aa7d-314a1e2de54a" containerName="kube-rbac-proxy-thanos"
Oct 30 16:48:25.269067 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.267925    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="66f44d22-9ae0-44aa-a585-cf1370bdb459" containerName="kube-rbac-proxy-web"
Oct 30 16:48:25.300300 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.300233    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.321079 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.321049    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 16:48:25.322285 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.322257    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 16:48:25.323847 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.323817    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 16:48:25.325034 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.325005    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6ig53cld0vd0\""
Oct 30 16:48:25.326458 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.326439    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 16:48:25.326547 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.326483    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 16:48:25.329360 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.329324    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 16:48:25.329481 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.329379    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 16:48:25.329481 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.329387    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 16:48:25.329481 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.329391    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 16:48:25.329481 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.329445    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 16:48:25.333431 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.333387    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 16:48:25.333431 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.333417    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 16:48:25.333566 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.333452    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 16:48:25.333599 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.333582    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 16:48:25.337163 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.337149    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rgj9h\""
Oct 30 16:48:25.337389 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.337375    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 16:48:25.443318 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.443276    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e10cae9c-9374-4615-8eda-d5d8d62173a3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.443318 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.443320    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e10cae9c-9374-4615-8eda-d5d8d62173a3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.443564 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.443340    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e10cae9c-9374-4615-8eda-d5d8d62173a3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.443564 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.443431    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e10cae9c-9374-4615-8eda-d5d8d62173a3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.443564 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.443526    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e10cae9c-9374-4615-8eda-d5d8d62173a3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.443564 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.443548    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e10cae9c-9374-4615-8eda-d5d8d62173a3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.443740 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.443567    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e10cae9c-9374-4615-8eda-d5d8d62173a3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.443740 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.443612    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e10cae9c-9374-4615-8eda-d5d8d62173a3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.443740 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.443655    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e10cae9c-9374-4615-8eda-d5d8d62173a3-web-config\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.443740 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.443679    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e10cae9c-9374-4615-8eda-d5d8d62173a3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.443740 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.443700    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e10cae9c-9374-4615-8eda-d5d8d62173a3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.443933 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.443762    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e10cae9c-9374-4615-8eda-d5d8d62173a3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.443933 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.443798    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e10cae9c-9374-4615-8eda-d5d8d62173a3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.443933 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.443832    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e10cae9c-9374-4615-8eda-d5d8d62173a3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.443933 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.443852    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e10cae9c-9374-4615-8eda-d5d8d62173a3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.443933 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.443872    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e10cae9c-9374-4615-8eda-d5d8d62173a3-config-out\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.443933 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.443889    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khwxs\" (UniqueName: \"kubernetes.io/projected/e10cae9c-9374-4615-8eda-d5d8d62173a3-kube-api-access-khwxs\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.444141 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.443939    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e10cae9c-9374-4615-8eda-d5d8d62173a3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.444141 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.444018    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e10cae9c-9374-4615-8eda-d5d8d62173a3-config\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.444141 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.444038    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e10cae9c-9374-4615-8eda-d5d8d62173a3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.544971 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.544931    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e10cae9c-9374-4615-8eda-d5d8d62173a3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.544971 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.544970    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e10cae9c-9374-4615-8eda-d5d8d62173a3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.545183 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.544991    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e10cae9c-9374-4615-8eda-d5d8d62173a3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.545183 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.545020    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e10cae9c-9374-4615-8eda-d5d8d62173a3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.545183 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.545049    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e10cae9c-9374-4615-8eda-d5d8d62173a3-config-out\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.545183 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.545075    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khwxs\" (UniqueName: \"kubernetes.io/projected/e10cae9c-9374-4615-8eda-d5d8d62173a3-kube-api-access-khwxs\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.545183 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.545108    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e10cae9c-9374-4615-8eda-d5d8d62173a3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.545183 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.545135    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e10cae9c-9374-4615-8eda-d5d8d62173a3-config\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.545183 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.545155    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e10cae9c-9374-4615-8eda-d5d8d62173a3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.545562 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.545202    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e10cae9c-9374-4615-8eda-d5d8d62173a3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.545562 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.545232    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e10cae9c-9374-4615-8eda-d5d8d62173a3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.545562 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.545263    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e10cae9c-9374-4615-8eda-d5d8d62173a3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.545562 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.545288    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e10cae9c-9374-4615-8eda-d5d8d62173a3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.545562 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.545326    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e10cae9c-9374-4615-8eda-d5d8d62173a3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.545562 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.545352    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e10cae9c-9374-4615-8eda-d5d8d62173a3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.545562 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.545382    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e10cae9c-9374-4615-8eda-d5d8d62173a3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.545562 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.545434    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e10cae9c-9374-4615-8eda-d5d8d62173a3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.545562 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.545470    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e10cae9c-9374-4615-8eda-d5d8d62173a3-web-config\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.545562 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.545493    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e10cae9c-9374-4615-8eda-d5d8d62173a3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.545562 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.545516    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e10cae9c-9374-4615-8eda-d5d8d62173a3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.546109 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.545865    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e10cae9c-9374-4615-8eda-d5d8d62173a3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.546109 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.545877    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e10cae9c-9374-4615-8eda-d5d8d62173a3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.546109 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.545896    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e10cae9c-9374-4615-8eda-d5d8d62173a3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.546109 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.545927    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e10cae9c-9374-4615-8eda-d5d8d62173a3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.546318 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.546190    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e10cae9c-9374-4615-8eda-d5d8d62173a3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.546771 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.546739    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e10cae9c-9374-4615-8eda-d5d8d62173a3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.549718 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.547483    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e10cae9c-9374-4615-8eda-d5d8d62173a3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.549718 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.548888    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e10cae9c-9374-4615-8eda-d5d8d62173a3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.549718 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.548935    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e10cae9c-9374-4615-8eda-d5d8d62173a3-config\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.549718 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.549392    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e10cae9c-9374-4615-8eda-d5d8d62173a3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.550545 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.550469    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e10cae9c-9374-4615-8eda-d5d8d62173a3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.550636 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.550616    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e10cae9c-9374-4615-8eda-d5d8d62173a3-config-out\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.550694 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.550660    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e10cae9c-9374-4615-8eda-d5d8d62173a3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.550694 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.550684    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e10cae9c-9374-4615-8eda-d5d8d62173a3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.550797 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.550692    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e10cae9c-9374-4615-8eda-d5d8d62173a3-web-config\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.550797 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.550708    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e10cae9c-9374-4615-8eda-d5d8d62173a3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.550941 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.550918    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e10cae9c-9374-4615-8eda-d5d8d62173a3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.551096 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.551081    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e10cae9c-9374-4615-8eda-d5d8d62173a3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.551853 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.551834    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e10cae9c-9374-4615-8eda-d5d8d62173a3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.570077 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.570041    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khwxs\" (UniqueName: \"kubernetes.io/projected/e10cae9c-9374-4615-8eda-d5d8d62173a3-kube-api-access-khwxs\") pod \"prometheus-k8s-0\" (UID: \"e10cae9c-9374-4615-8eda-d5d8d62173a3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:25.644343 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.644302    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5584db78d6-2wdm4"]
Oct 30 16:48:25.692798 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.692765    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dcced64-e20a-4247-aa7d-314a1e2de54a" path="/var/lib/kubelet/pods/0dcced64-e20a-4247-aa7d-314a1e2de54a/volumes"
Oct 30 16:48:25.709094 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:25.709064    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:26.077236 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:26.077190    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 16:48:26.080089 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:48:26.080036    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode10cae9c_9374_4615_8eda_d5d8d62173a3.slice/crio-bab98272bc71a72fa2c7897ddf690fe9b3efaa23941114874ec5e9c27c2cfd1d WatchSource:0}: Error finding container bab98272bc71a72fa2c7897ddf690fe9b3efaa23941114874ec5e9c27c2cfd1d: Status 404 returned error can't find the container with id bab98272bc71a72fa2c7897ddf690fe9b3efaa23941114874ec5e9c27c2cfd1d
Oct 30 16:48:26.178782 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:26.178750    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e10cae9c-9374-4615-8eda-d5d8d62173a3","Type":"ContainerStarted","Data":"ae468522224e62de2db2e60b3f2da406a497b929008be14682727467cfc1f6d9"}
Oct 30 16:48:26.178913 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:26.178813    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e10cae9c-9374-4615-8eda-d5d8d62173a3","Type":"ContainerStarted","Data":"bab98272bc71a72fa2c7897ddf690fe9b3efaa23941114874ec5e9c27c2cfd1d"}
Oct 30 16:48:26.975579 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:48:26.975537    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 16:48:27.183617 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:27.183578    4433 generic.go:356] "Generic (PLEG): container finished" podID="e10cae9c-9374-4615-8eda-d5d8d62173a3" containerID="ae468522224e62de2db2e60b3f2da406a497b929008be14682727467cfc1f6d9" exitCode=0
Oct 30 16:48:27.184024 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:27.183668    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e10cae9c-9374-4615-8eda-d5d8d62173a3","Type":"ContainerDied","Data":"ae468522224e62de2db2e60b3f2da406a497b929008be14682727467cfc1f6d9"}
Oct 30 16:48:28.190595 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:28.190554    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e10cae9c-9374-4615-8eda-d5d8d62173a3","Type":"ContainerStarted","Data":"ca1d7494fb0863b5d7ec37efc98817bcd0039067ce351f8789a29c48875ae2ef"}
Oct 30 16:48:28.191054 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:28.190637    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e10cae9c-9374-4615-8eda-d5d8d62173a3","Type":"ContainerStarted","Data":"db210ca50e23474337198d58ae043ccbbe5ea1c06b5e438e555b1ae00a692ad3"}
Oct 30 16:48:28.191054 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:28.190678    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e10cae9c-9374-4615-8eda-d5d8d62173a3","Type":"ContainerStarted","Data":"03075c280769743f13e7505b14cb97e1d8a40c19b5347e2149d0da31c9c09eb6"}
Oct 30 16:48:28.191054 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:28.190720    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e10cae9c-9374-4615-8eda-d5d8d62173a3","Type":"ContainerStarted","Data":"e47fc5a564c14a3f358419741df95f5283a44806b6c305c7a8e78872fde5ab93"}
Oct 30 16:48:28.191054 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:28.190755    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e10cae9c-9374-4615-8eda-d5d8d62173a3","Type":"ContainerStarted","Data":"57ecd663a01e49c57551d6d926bcfe1e196f6f10f4446d99ba5ee205389386b3"}
Oct 30 16:48:28.191054 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:28.190789    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e10cae9c-9374-4615-8eda-d5d8d62173a3","Type":"ContainerStarted","Data":"7aa21f14a5ef3dd1cf97db2878661ab483a8842197f28c56e5c936cb3d827719"}
Oct 30 16:48:28.273528 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:28.273468    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.254852022 podStartE2EDuration="3.273453115s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.601093ms" isStatefulPod=false podCreationTimestamp="2026-10-30 16:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:48:28.261540959 +0000 UTC m=+84.696443166" watchObservedRunningTime="2026-10-30 16:48:28.273453115 +0000 UTC m=+84.708355317"
Oct 30 16:48:30.709814 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:30.709778    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:48:50.663980 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:50.663906    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5584db78d6-2wdm4" podUID="914e3790-ef93-47ee-90e9-419cc4a7f3ca" containerName="console" containerID="cri-o://13f18921aa9bad120c55a357c550659ab8b71f5734a25ad6bcd74896c371f62a" gracePeriod=15
Oct 30 16:48:50.904391 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:50.904365    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5584db78d6-2wdm4"
Oct 30 16:48:50.971913 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:50.971879    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/914e3790-ef93-47ee-90e9-419cc4a7f3ca-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/914e3790-ef93-47ee-90e9-419cc4a7f3ca-console-oauth-config\") pod \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\" (UID: \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\") "
Oct 30 16:48:50.972096 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:50.971919    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/914e3790-ef93-47ee-90e9-419cc4a7f3ca-console-config\" (UniqueName: \"kubernetes.io/configmap/914e3790-ef93-47ee-90e9-419cc4a7f3ca-console-config\") pod \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\" (UID: \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\") "
Oct 30 16:48:50.972096 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:50.971941    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/914e3790-ef93-47ee-90e9-419cc4a7f3ca-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/914e3790-ef93-47ee-90e9-419cc4a7f3ca-trusted-ca-bundle\") pod \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\" (UID: \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\") "
Oct 30 16:48:50.972096 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:50.971979    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/914e3790-ef93-47ee-90e9-419cc4a7f3ca-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/914e3790-ef93-47ee-90e9-419cc4a7f3ca-oauth-serving-cert\") pod \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\" (UID: \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\") "
Oct 30 16:48:50.972096 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:50.972015    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/914e3790-ef93-47ee-90e9-419cc4a7f3ca-service-ca\" (UniqueName: \"kubernetes.io/configmap/914e3790-ef93-47ee-90e9-419cc4a7f3ca-service-ca\") pod \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\" (UID: \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\") "
Oct 30 16:48:50.972096 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:50.972061    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914e3790-ef93-47ee-90e9-419cc4a7f3ca-tmp\" (UniqueName: \"kubernetes.io/empty-dir/914e3790-ef93-47ee-90e9-419cc4a7f3ca-tmp\") pod \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\" (UID: \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\") "
Oct 30 16:48:50.972366 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:50.972097    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/914e3790-ef93-47ee-90e9-419cc4a7f3ca-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/914e3790-ef93-47ee-90e9-419cc4a7f3ca-console-serving-cert\") pod \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\" (UID: \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\") "
Oct 30 16:48:50.972366 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:50.972136    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/914e3790-ef93-47ee-90e9-419cc4a7f3ca-kube-api-access-75pps\" (UniqueName: \"kubernetes.io/projected/914e3790-ef93-47ee-90e9-419cc4a7f3ca-kube-api-access-75pps\") pod \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\" (UID: \"914e3790-ef93-47ee-90e9-419cc4a7f3ca\") "
Oct 30 16:48:50.972504 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:50.972432    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914e3790-ef93-47ee-90e9-419cc4a7f3ca-tmp" pod "914e3790-ef93-47ee-90e9-419cc4a7f3ca" (UID: "914e3790-ef93-47ee-90e9-419cc4a7f3ca"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:48:50.972504 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:50.972465    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/914e3790-ef93-47ee-90e9-419cc4a7f3ca-oauth-serving-cert" pod "914e3790-ef93-47ee-90e9-419cc4a7f3ca" (UID: "914e3790-ef93-47ee-90e9-419cc4a7f3ca"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:48:50.972504 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:50.972481    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/914e3790-ef93-47ee-90e9-419cc4a7f3ca-trusted-ca-bundle" pod "914e3790-ef93-47ee-90e9-419cc4a7f3ca" (UID: "914e3790-ef93-47ee-90e9-419cc4a7f3ca"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:48:50.972504 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:50.972494    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/914e3790-ef93-47ee-90e9-419cc4a7f3ca-console-config" pod "914e3790-ef93-47ee-90e9-419cc4a7f3ca" (UID: "914e3790-ef93-47ee-90e9-419cc4a7f3ca"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:48:50.972865 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:50.972837    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/914e3790-ef93-47ee-90e9-419cc4a7f3ca-service-ca" pod "914e3790-ef93-47ee-90e9-419cc4a7f3ca" (UID: "914e3790-ef93-47ee-90e9-419cc4a7f3ca"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:48:50.974279 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:50.974249    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/914e3790-ef93-47ee-90e9-419cc4a7f3ca-console-oauth-config" pod "914e3790-ef93-47ee-90e9-419cc4a7f3ca" (UID: "914e3790-ef93-47ee-90e9-419cc4a7f3ca"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:48:50.974462 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:50.974367    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/914e3790-ef93-47ee-90e9-419cc4a7f3ca-console-serving-cert" pod "914e3790-ef93-47ee-90e9-419cc4a7f3ca" (UID: "914e3790-ef93-47ee-90e9-419cc4a7f3ca"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:48:50.974709 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:50.974691    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/914e3790-ef93-47ee-90e9-419cc4a7f3ca-kube-api-access-75pps" pod "914e3790-ef93-47ee-90e9-419cc4a7f3ca" (UID: "914e3790-ef93-47ee-90e9-419cc4a7f3ca"). InnerVolumeSpecName "kube-api-access-75pps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:48:51.073011 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:51.072972    4433 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/914e3790-ef93-47ee-90e9-419cc4a7f3ca-console-oauth-config\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:51.073011 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:51.073002    4433 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/914e3790-ef93-47ee-90e9-419cc4a7f3ca-console-config\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:51.073011 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:51.073011    4433 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/914e3790-ef93-47ee-90e9-419cc4a7f3ca-trusted-ca-bundle\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:51.073011 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:51.073020    4433 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/914e3790-ef93-47ee-90e9-419cc4a7f3ca-oauth-serving-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:51.073250 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:51.073028    4433 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/914e3790-ef93-47ee-90e9-419cc4a7f3ca-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:51.073250 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:51.073036    4433 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/914e3790-ef93-47ee-90e9-419cc4a7f3ca-tmp\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:51.073250 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:51.073044    4433 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/914e3790-ef93-47ee-90e9-419cc4a7f3ca-console-serving-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:51.073250 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:51.073053    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75pps\" (UniqueName: \"kubernetes.io/projected/914e3790-ef93-47ee-90e9-419cc4a7f3ca-kube-api-access-75pps\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:48:51.275137 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:51.275046    4433 generic.go:356] "Generic (PLEG): container finished" podID="914e3790-ef93-47ee-90e9-419cc4a7f3ca" containerID="13f18921aa9bad120c55a357c550659ab8b71f5734a25ad6bcd74896c371f62a" exitCode=0
Oct 30 16:48:51.275137 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:51.275093    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5584db78d6-2wdm4"
Oct 30 16:48:51.275137 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:51.275126    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5584db78d6-2wdm4" event={"ID":"914e3790-ef93-47ee-90e9-419cc4a7f3ca","Type":"ContainerDied","Data":"13f18921aa9bad120c55a357c550659ab8b71f5734a25ad6bcd74896c371f62a"}
Oct 30 16:48:51.275397 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:51.275176    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5584db78d6-2wdm4" event={"ID":"914e3790-ef93-47ee-90e9-419cc4a7f3ca","Type":"ContainerDied","Data":"42233afa3cac1f09786ffedb28ef27ab35043df29c8eb6a4ea22819f2f4e14c4"}
Oct 30 16:48:51.275397 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:51.275202    4433 scope.go:122] "RemoveContainer" containerID="13f18921aa9bad120c55a357c550659ab8b71f5734a25ad6bcd74896c371f62a"
Oct 30 16:48:51.283810 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:51.283785    4433 scope.go:122] "RemoveContainer" containerID="13f18921aa9bad120c55a357c550659ab8b71f5734a25ad6bcd74896c371f62a"
Oct 30 16:48:51.284068 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:48:51.284048    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13f18921aa9bad120c55a357c550659ab8b71f5734a25ad6bcd74896c371f62a\": container with ID starting with 13f18921aa9bad120c55a357c550659ab8b71f5734a25ad6bcd74896c371f62a not found: ID does not exist" containerID="13f18921aa9bad120c55a357c550659ab8b71f5734a25ad6bcd74896c371f62a"
Oct 30 16:48:51.284117 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:51.284077    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13f18921aa9bad120c55a357c550659ab8b71f5734a25ad6bcd74896c371f62a"} err="failed to get container status \"13f18921aa9bad120c55a357c550659ab8b71f5734a25ad6bcd74896c371f62a\": rpc error: code = NotFound desc = could not find container \"13f18921aa9bad120c55a357c550659ab8b71f5734a25ad6bcd74896c371f62a\": container with ID starting with 13f18921aa9bad120c55a357c550659ab8b71f5734a25ad6bcd74896c371f62a not found: ID does not exist"
Oct 30 16:48:51.323936 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:51.323875    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5584db78d6-2wdm4"]
Oct 30 16:48:51.335201 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:51.335178    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5584db78d6-2wdm4"]
Oct 30 16:48:51.691915 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:48:51.691886    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="914e3790-ef93-47ee-90e9-419cc4a7f3ca" path="/var/lib/kubelet/pods/914e3790-ef93-47ee-90e9-419cc4a7f3ca/volumes"
Oct 30 16:48:56.975201 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:48:56.975161    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 16:49:25.710218 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:49:25.710182    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:49:25.725864 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:49:25.725835    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:49:26.409114 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:49:26.409084    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:49:26.976859 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:49:26.976822    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 16:49:56.975762 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:49:56.975724    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 16:50:26.977216 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:50:26.977175    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 16:50:56.976228 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:50:56.976152    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 16:51:09.921333 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:09.921271    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-bqbpc"]
Oct 30 16:51:09.921867 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:09.921657    4433 cpu_manager.go:414] "containerMap: removing container" podUID="914e3790-ef93-47ee-90e9-419cc4a7f3ca" containerName="console"
Oct 30 16:51:09.921867 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:09.921672    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="914e3790-ef93-47ee-90e9-419cc4a7f3ca" containerName="console"
Oct 30 16:51:09.921867 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:09.921728    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="914e3790-ef93-47ee-90e9-419cc4a7f3ca" containerName="console"
Oct 30 16:51:09.924665 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:09.924646    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-bqbpc"
Oct 30 16:51:09.950663 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:09.950634    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 30 16:51:09.950805 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:09.950681    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-ff29l\""
Oct 30 16:51:09.950805 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:09.950785    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/615201d1-7098-46ad-b5c4-4c0863b0c7cd-tmp\") pod \"openshift-lws-operator-8465994787-bqbpc\" (UID: \"615201d1-7098-46ad-b5c4-4c0863b0c7cd\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-bqbpc"
Oct 30 16:51:09.950881 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:09.950813    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 30 16:51:09.950916 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:09.950887    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkxqp\" (UniqueName: \"kubernetes.io/projected/615201d1-7098-46ad-b5c4-4c0863b0c7cd-kube-api-access-qkxqp\") pod \"openshift-lws-operator-8465994787-bqbpc\" (UID: \"615201d1-7098-46ad-b5c4-4c0863b0c7cd\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-bqbpc"
Oct 30 16:51:09.951852 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:09.951836    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-bqbpc"]
Oct 30 16:51:10.051813 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:10.051773    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/615201d1-7098-46ad-b5c4-4c0863b0c7cd-tmp\") pod \"openshift-lws-operator-8465994787-bqbpc\" (UID: \"615201d1-7098-46ad-b5c4-4c0863b0c7cd\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-bqbpc"
Oct 30 16:51:10.051991 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:10.051855    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkxqp\" (UniqueName: \"kubernetes.io/projected/615201d1-7098-46ad-b5c4-4c0863b0c7cd-kube-api-access-qkxqp\") pod \"openshift-lws-operator-8465994787-bqbpc\" (UID: \"615201d1-7098-46ad-b5c4-4c0863b0c7cd\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-bqbpc"
Oct 30 16:51:10.052183 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:10.052160    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/615201d1-7098-46ad-b5c4-4c0863b0c7cd-tmp\") pod \"openshift-lws-operator-8465994787-bqbpc\" (UID: \"615201d1-7098-46ad-b5c4-4c0863b0c7cd\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-bqbpc"
Oct 30 16:51:10.075309 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:10.075278    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkxqp\" (UniqueName: \"kubernetes.io/projected/615201d1-7098-46ad-b5c4-4c0863b0c7cd-kube-api-access-qkxqp\") pod \"openshift-lws-operator-8465994787-bqbpc\" (UID: \"615201d1-7098-46ad-b5c4-4c0863b0c7cd\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-bqbpc"
Oct 30 16:51:10.248627 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:10.248587    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-bqbpc"
Oct 30 16:51:10.542386 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:10.542298    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-bqbpc"]
Oct 30 16:51:10.547006 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:51:10.546981    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod615201d1_7098_46ad_b5c4_4c0863b0c7cd.slice/crio-73555747e1ec82f06d5e0fbe1dccb26c0f040f68b115571e7bc6dc2993cbc6ff WatchSource:0}: Error finding container 73555747e1ec82f06d5e0fbe1dccb26c0f040f68b115571e7bc6dc2993cbc6ff: Status 404 returned error can't find the container with id 73555747e1ec82f06d5e0fbe1dccb26c0f040f68b115571e7bc6dc2993cbc6ff
Oct 30 16:51:10.752394 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:10.752362    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-bqbpc" event={"ID":"615201d1-7098-46ad-b5c4-4c0863b0c7cd","Type":"ContainerStarted","Data":"73555747e1ec82f06d5e0fbe1dccb26c0f040f68b115571e7bc6dc2993cbc6ff"}
Oct 30 16:51:13.765105 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:13.765072    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-bqbpc" event={"ID":"615201d1-7098-46ad-b5c4-4c0863b0c7cd","Type":"ContainerStarted","Data":"eb12a64391ec8e0253d02782f7005ed057dce86fb4f96f9a04c472169972524c"}
Oct 30 16:51:13.805513 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:13.805470    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-8465994787-bqbpc" podStartSLOduration=2.113833664 podStartE2EDuration="4.805456363s" totalImagesPullingTime="2.691622699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:13.80461401 +0000 UTC m=+250.239516225" watchObservedRunningTime="2026-10-30 16:51:13.805456363 +0000 UTC m=+250.240358569"
Oct 30 16:51:26.976805 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:51:26.976769    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 16:51:28.160522 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:28.160488    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68c46f97f5-wpvf2"]
Oct 30 16:51:28.163749 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:28.163732    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-wpvf2"
Oct 30 16:51:28.183291 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:28.183264    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f08eb4e1-c182-499c-8fa3-d8e57cdebaeb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68c46f97f5-wpvf2\" (UID: \"f08eb4e1-c182-499c-8fa3-d8e57cdebaeb\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-wpvf2"
Oct 30 16:51:28.183417 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:28.183300    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f08eb4e1-c182-499c-8fa3-d8e57cdebaeb-tmp\") pod \"opendatahub-operator-controller-manager-68c46f97f5-wpvf2\" (UID: \"f08eb4e1-c182-499c-8fa3-d8e57cdebaeb\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-wpvf2"
Oct 30 16:51:28.183417 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:28.183323    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f08eb4e1-c182-499c-8fa3-d8e57cdebaeb-metrics-cert\") pod \"opendatahub-operator-controller-manager-68c46f97f5-wpvf2\" (UID: \"f08eb4e1-c182-499c-8fa3-d8e57cdebaeb\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-wpvf2"
Oct 30 16:51:28.183417 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:28.183386    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tlnx\" (UniqueName: \"kubernetes.io/projected/f08eb4e1-c182-499c-8fa3-d8e57cdebaeb-kube-api-access-5tlnx\") pod \"opendatahub-operator-controller-manager-68c46f97f5-wpvf2\" (UID: \"f08eb4e1-c182-499c-8fa3-d8e57cdebaeb\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-wpvf2"
Oct 30 16:51:28.183532 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:28.183458    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f08eb4e1-c182-499c-8fa3-d8e57cdebaeb-webhook-cert\") pod \"opendatahub-operator-controller-manager-68c46f97f5-wpvf2\" (UID: \"f08eb4e1-c182-499c-8fa3-d8e57cdebaeb\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-wpvf2"
Oct 30 16:51:28.183532 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:28.183484    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f08eb4e1-c182-499c-8fa3-d8e57cdebaeb-manifests\") pod \"opendatahub-operator-controller-manager-68c46f97f5-wpvf2\" (UID: \"f08eb4e1-c182-499c-8fa3-d8e57cdebaeb\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-wpvf2"
Oct 30 16:51:28.195838 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:28.195810    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68c46f97f5-wpvf2"]
Oct 30 16:51:28.265155 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:28.265122    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 16:51:28.265495 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:28.265479    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 16:51:28.283897 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:28.283875    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f08eb4e1-c182-499c-8fa3-d8e57cdebaeb-webhook-cert\") pod \"opendatahub-operator-controller-manager-68c46f97f5-wpvf2\" (UID: \"f08eb4e1-c182-499c-8fa3-d8e57cdebaeb\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-wpvf2"
Oct 30 16:51:28.283950 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:28.283904    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f08eb4e1-c182-499c-8fa3-d8e57cdebaeb-manifests\") pod \"opendatahub-operator-controller-manager-68c46f97f5-wpvf2\" (UID: \"f08eb4e1-c182-499c-8fa3-d8e57cdebaeb\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-wpvf2"
Oct 30 16:51:28.283950 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:28.283942    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f08eb4e1-c182-499c-8fa3-d8e57cdebaeb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68c46f97f5-wpvf2\" (UID: \"f08eb4e1-c182-499c-8fa3-d8e57cdebaeb\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-wpvf2"
Oct 30 16:51:28.284110 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:28.284089    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f08eb4e1-c182-499c-8fa3-d8e57cdebaeb-tmp\") pod \"opendatahub-operator-controller-manager-68c46f97f5-wpvf2\" (UID: \"f08eb4e1-c182-499c-8fa3-d8e57cdebaeb\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-wpvf2"
Oct 30 16:51:28.284166 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:28.284146    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f08eb4e1-c182-499c-8fa3-d8e57cdebaeb-metrics-cert\") pod \"opendatahub-operator-controller-manager-68c46f97f5-wpvf2\" (UID: \"f08eb4e1-c182-499c-8fa3-d8e57cdebaeb\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-wpvf2"
Oct 30 16:51:28.284202 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:28.284185    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tlnx\" (UniqueName: \"kubernetes.io/projected/f08eb4e1-c182-499c-8fa3-d8e57cdebaeb-kube-api-access-5tlnx\") pod \"opendatahub-operator-controller-manager-68c46f97f5-wpvf2\" (UID: \"f08eb4e1-c182-499c-8fa3-d8e57cdebaeb\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-wpvf2"
Oct 30 16:51:28.284266 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:28.284246    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vn468\""
Oct 30 16:51:28.284441 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:28.284394    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f08eb4e1-c182-499c-8fa3-d8e57cdebaeb-manifests\") pod \"opendatahub-operator-controller-manager-68c46f97f5-wpvf2\" (UID: \"f08eb4e1-c182-499c-8fa3-d8e57cdebaeb\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-wpvf2"
Oct 30 16:51:28.284783 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:28.284757    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Oct 30 16:51:28.285072 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:28.284818    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 16:51:28.285072 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:28.284964    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 16:51:28.286519 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:28.286491    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f08eb4e1-c182-499c-8fa3-d8e57cdebaeb-tmp\") pod \"opendatahub-operator-controller-manager-68c46f97f5-wpvf2\" (UID: \"f08eb4e1-c182-499c-8fa3-d8e57cdebaeb\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-wpvf2"
Oct 30 16:51:28.286634 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:28.286620    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f08eb4e1-c182-499c-8fa3-d8e57cdebaeb-webhook-cert\") pod \"opendatahub-operator-controller-manager-68c46f97f5-wpvf2\" (UID: \"f08eb4e1-c182-499c-8fa3-d8e57cdebaeb\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-wpvf2"
Oct 30 16:51:28.286697 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:28.286671    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f08eb4e1-c182-499c-8fa3-d8e57cdebaeb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68c46f97f5-wpvf2\" (UID: \"f08eb4e1-c182-499c-8fa3-d8e57cdebaeb\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-wpvf2"
Oct 30 16:51:28.296603 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:28.296577    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f08eb4e1-c182-499c-8fa3-d8e57cdebaeb-metrics-cert\") pod \"opendatahub-operator-controller-manager-68c46f97f5-wpvf2\" (UID: \"f08eb4e1-c182-499c-8fa3-d8e57cdebaeb\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-wpvf2"
Oct 30 16:51:28.312157 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:28.312130    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tlnx\" (UniqueName: \"kubernetes.io/projected/f08eb4e1-c182-499c-8fa3-d8e57cdebaeb-kube-api-access-5tlnx\") pod \"opendatahub-operator-controller-manager-68c46f97f5-wpvf2\" (UID: \"f08eb4e1-c182-499c-8fa3-d8e57cdebaeb\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-wpvf2"
Oct 30 16:51:28.478020 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:28.477986    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-wpvf2"
Oct 30 16:51:28.834548 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:28.834474    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68c46f97f5-wpvf2"]
Oct 30 16:51:28.837691 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:51:28.837665    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf08eb4e1_c182_499c_8fa3_d8e57cdebaeb.slice/crio-4cf05b298ecaa13e205de7bd6cb1b9dc19b4b2e6cb94d2fe6f5a0dfc2948e690 WatchSource:0}: Error finding container 4cf05b298ecaa13e205de7bd6cb1b9dc19b4b2e6cb94d2fe6f5a0dfc2948e690: Status 404 returned error can't find the container with id 4cf05b298ecaa13e205de7bd6cb1b9dc19b4b2e6cb94d2fe6f5a0dfc2948e690
Oct 30 16:51:29.829875 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:29.829828    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-wpvf2" event={"ID":"f08eb4e1-c182-499c-8fa3-d8e57cdebaeb","Type":"ContainerStarted","Data":"4cf05b298ecaa13e205de7bd6cb1b9dc19b4b2e6cb94d2fe6f5a0dfc2948e690"}
Oct 30 16:51:31.839747 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:31.839710    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-wpvf2" event={"ID":"f08eb4e1-c182-499c-8fa3-d8e57cdebaeb","Type":"ContainerStarted","Data":"1d24cbc754a66639d3b960d64528e48409597eeb61bc3c29b5ba50370d767319"}
Oct 30 16:51:32.847716 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:32.847680    4433 generic.go:356] "Generic (PLEG): container finished" podID="f08eb4e1-c182-499c-8fa3-d8e57cdebaeb" containerID="1d24cbc754a66639d3b960d64528e48409597eeb61bc3c29b5ba50370d767319" exitCode=0
Oct 30 16:51:32.848180 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:32.847762    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-wpvf2" event={"ID":"f08eb4e1-c182-499c-8fa3-d8e57cdebaeb","Type":"ContainerDied","Data":"1d24cbc754a66639d3b960d64528e48409597eeb61bc3c29b5ba50370d767319"}
Oct 30 16:51:33.858414 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:33.858354    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-wpvf2" event={"ID":"f08eb4e1-c182-499c-8fa3-d8e57cdebaeb","Type":"ContainerStarted","Data":"044e16e166698fe334f1deb60f0f3daaca6b5bf4efb5eaa6b0bd443f0115c917"}
Oct 30 16:51:33.858869 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:33.858479    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-wpvf2"
Oct 30 16:51:33.944357 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:33.944298    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-wpvf2" podStartSLOduration=2.260162618 podStartE2EDuration="5.944284279s" totalImagesPullingTime="3.222891777s" totalInitContainerRuntime="461.229884ms" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:33.932018072 +0000 UTC m=+270.366920278" watchObservedRunningTime="2026-10-30 16:51:33.944284279 +0000 UTC m=+270.379186485"
Oct 30 16:51:44.867826 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:44.867780    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-wpvf2"
Oct 30 16:51:49.923861 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:49.923826    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-fc9769c4f-f42wp"]
Oct 30 16:51:49.932744 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:49.932715    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-f42wp"
Oct 30 16:51:49.959105 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:49.959080    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Oct 30 16:51:49.959253 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:49.959119    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Oct 30 16:51:49.959253 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:49.959187    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-htr64\""
Oct 30 16:51:49.959337 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:49.959314    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Oct 30 16:51:49.960570 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:49.960543    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-fc9769c4f-f42wp"]
Oct 30 16:51:50.065068 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:50.065019    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44f84f47-d0a0-4ad9-9d75-1794e6c718d0-cert\") pod \"lws-controller-manager-fc9769c4f-f42wp\" (UID: \"44f84f47-d0a0-4ad9-9d75-1794e6c718d0\") " pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-f42wp"
Oct 30 16:51:50.065265 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:50.065079    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dll9\" (UniqueName: \"kubernetes.io/projected/44f84f47-d0a0-4ad9-9d75-1794e6c718d0-kube-api-access-7dll9\") pod \"lws-controller-manager-fc9769c4f-f42wp\" (UID: \"44f84f47-d0a0-4ad9-9d75-1794e6c718d0\") " pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-f42wp"
Oct 30 16:51:50.065265 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:50.065112    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/44f84f47-d0a0-4ad9-9d75-1794e6c718d0-metrics-cert\") pod \"lws-controller-manager-fc9769c4f-f42wp\" (UID: \"44f84f47-d0a0-4ad9-9d75-1794e6c718d0\") " pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-f42wp"
Oct 30 16:51:50.065394 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:50.065277    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/44f84f47-d0a0-4ad9-9d75-1794e6c718d0-manager-config\") pod \"lws-controller-manager-fc9769c4f-f42wp\" (UID: \"44f84f47-d0a0-4ad9-9d75-1794e6c718d0\") " pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-f42wp"
Oct 30 16:51:50.166566 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:50.166528    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/44f84f47-d0a0-4ad9-9d75-1794e6c718d0-manager-config\") pod \"lws-controller-manager-fc9769c4f-f42wp\" (UID: \"44f84f47-d0a0-4ad9-9d75-1794e6c718d0\") " pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-f42wp"
Oct 30 16:51:50.166754 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:50.166602    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44f84f47-d0a0-4ad9-9d75-1794e6c718d0-cert\") pod \"lws-controller-manager-fc9769c4f-f42wp\" (UID: \"44f84f47-d0a0-4ad9-9d75-1794e6c718d0\") " pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-f42wp"
Oct 30 16:51:50.166754 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:50.166633    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dll9\" (UniqueName: \"kubernetes.io/projected/44f84f47-d0a0-4ad9-9d75-1794e6c718d0-kube-api-access-7dll9\") pod \"lws-controller-manager-fc9769c4f-f42wp\" (UID: \"44f84f47-d0a0-4ad9-9d75-1794e6c718d0\") " pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-f42wp"
Oct 30 16:51:50.166754 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:50.166662    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/44f84f47-d0a0-4ad9-9d75-1794e6c718d0-metrics-cert\") pod \"lws-controller-manager-fc9769c4f-f42wp\" (UID: \"44f84f47-d0a0-4ad9-9d75-1794e6c718d0\") " pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-f42wp"
Oct 30 16:51:50.167326 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:50.167302    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/44f84f47-d0a0-4ad9-9d75-1794e6c718d0-manager-config\") pod \"lws-controller-manager-fc9769c4f-f42wp\" (UID: \"44f84f47-d0a0-4ad9-9d75-1794e6c718d0\") " pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-f42wp"
Oct 30 16:51:50.169234 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:50.169210    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/44f84f47-d0a0-4ad9-9d75-1794e6c718d0-metrics-cert\") pod \"lws-controller-manager-fc9769c4f-f42wp\" (UID: \"44f84f47-d0a0-4ad9-9d75-1794e6c718d0\") " pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-f42wp"
Oct 30 16:51:50.169381 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:50.169243    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44f84f47-d0a0-4ad9-9d75-1794e6c718d0-cert\") pod \"lws-controller-manager-fc9769c4f-f42wp\" (UID: \"44f84f47-d0a0-4ad9-9d75-1794e6c718d0\") " pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-f42wp"
Oct 30 16:51:50.189745 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:50.189715    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dll9\" (UniqueName: \"kubernetes.io/projected/44f84f47-d0a0-4ad9-9d75-1794e6c718d0-kube-api-access-7dll9\") pod \"lws-controller-manager-fc9769c4f-f42wp\" (UID: \"44f84f47-d0a0-4ad9-9d75-1794e6c718d0\") " pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-f42wp"
Oct 30 16:51:50.244320 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:50.244284    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-f42wp"
Oct 30 16:51:50.537855 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:50.537774    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-fc9769c4f-f42wp"]
Oct 30 16:51:50.541787 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:51:50.541754    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44f84f47_d0a0_4ad9_9d75_1794e6c718d0.slice/crio-968d64205534a4f20982b503f4604fdcd8673f1076a1c71e9e9fac56ce4be2b5 WatchSource:0}: Error finding container 968d64205534a4f20982b503f4604fdcd8673f1076a1c71e9e9fac56ce4be2b5: Status 404 returned error can't find the container with id 968d64205534a4f20982b503f4604fdcd8673f1076a1c71e9e9fac56ce4be2b5
Oct 30 16:51:50.928724 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:50.928691    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-f42wp" event={"ID":"44f84f47-d0a0-4ad9-9d75-1794e6c718d0","Type":"ContainerStarted","Data":"968d64205534a4f20982b503f4604fdcd8673f1076a1c71e9e9fac56ce4be2b5"}
Oct 30 16:51:51.808629 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:51.808574    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-observability-584f99c788-j2dld"]
Oct 30 16:51:51.812241 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:51.812217    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-j2dld"
Oct 30 16:51:51.829612 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:51.829585    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-webhook-cert\""
Oct 30 16:51:51.841432 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:51.841383    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-dockercfg-sh546\""
Oct 30 16:51:51.842896 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:51.842871    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-j2dld"]
Oct 30 16:51:51.881008 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:51.880968    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzmj8\" (UniqueName: \"kubernetes.io/projected/8ebb4433-b279-485f-b5c8-7b72cefaec9e-kube-api-access-nzmj8\") pod \"odh-observability-584f99c788-j2dld\" (UID: \"8ebb4433-b279-485f-b5c8-7b72cefaec9e\") " pod="opendatahub/odh-observability-584f99c788-j2dld"
Oct 30 16:51:51.881203 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:51.881023    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/8ebb4433-b279-485f-b5c8-7b72cefaec9e-webhook-certs\") pod \"odh-observability-584f99c788-j2dld\" (UID: \"8ebb4433-b279-485f-b5c8-7b72cefaec9e\") " pod="opendatahub/odh-observability-584f99c788-j2dld"
Oct 30 16:51:51.982448 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:51.982389    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzmj8\" (UniqueName: \"kubernetes.io/projected/8ebb4433-b279-485f-b5c8-7b72cefaec9e-kube-api-access-nzmj8\") pod \"odh-observability-584f99c788-j2dld\" (UID: \"8ebb4433-b279-485f-b5c8-7b72cefaec9e\") " pod="opendatahub/odh-observability-584f99c788-j2dld"
Oct 30 16:51:51.982924 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:51.982460    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/8ebb4433-b279-485f-b5c8-7b72cefaec9e-webhook-certs\") pod \"odh-observability-584f99c788-j2dld\" (UID: \"8ebb4433-b279-485f-b5c8-7b72cefaec9e\") " pod="opendatahub/odh-observability-584f99c788-j2dld"
Oct 30 16:51:51.982924 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:51:51.982622    4433 secret.go:189] Couldn't get secret opendatahub/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found
Oct 30 16:51:51.982924 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:51:51.982695    4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ebb4433-b279-485f-b5c8-7b72cefaec9e-webhook-certs podName:8ebb4433-b279-485f-b5c8-7b72cefaec9e nodeName:}" failed. No retries permitted until 2026-10-30 16:51:52.482676349 +0000 UTC m=+288.917578549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/8ebb4433-b279-485f-b5c8-7b72cefaec9e-webhook-certs") pod "odh-observability-584f99c788-j2dld" (UID: "8ebb4433-b279-485f-b5c8-7b72cefaec9e") : secret "odh-observability-webhook-cert" not found
Oct 30 16:51:52.006374 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:52.006339    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzmj8\" (UniqueName: \"kubernetes.io/projected/8ebb4433-b279-485f-b5c8-7b72cefaec9e-kube-api-access-nzmj8\") pod \"odh-observability-584f99c788-j2dld\" (UID: \"8ebb4433-b279-485f-b5c8-7b72cefaec9e\") " pod="opendatahub/odh-observability-584f99c788-j2dld"
Oct 30 16:51:52.487755 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:52.487716    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/8ebb4433-b279-485f-b5c8-7b72cefaec9e-webhook-certs\") pod \"odh-observability-584f99c788-j2dld\" (UID: \"8ebb4433-b279-485f-b5c8-7b72cefaec9e\") " pod="opendatahub/odh-observability-584f99c788-j2dld"
Oct 30 16:51:52.490190 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:52.490167    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/8ebb4433-b279-485f-b5c8-7b72cefaec9e-webhook-certs\") pod \"odh-observability-584f99c788-j2dld\" (UID: \"8ebb4433-b279-485f-b5c8-7b72cefaec9e\") " pod="opendatahub/odh-observability-584f99c788-j2dld"
Oct 30 16:51:52.728667 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:52.728620    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-j2dld"
Oct 30 16:51:52.938152 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:52.938112    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-f42wp" event={"ID":"44f84f47-d0a0-4ad9-9d75-1794e6c718d0","Type":"ContainerStarted","Data":"fcf0fb1c4b4f9c2de6e227e3798b397048d787a6ff851de52dca66e742e0419d"}
Oct 30 16:51:52.938350 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:52.938212    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-f42wp"
Oct 30 16:51:52.979584 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:52.979522    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-f42wp" podStartSLOduration=2.377147998 podStartE2EDuration="3.97950093s" totalImagesPullingTime="1.602352932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:52.978311424 +0000 UTC m=+289.413213630" watchObservedRunningTime="2026-10-30 16:51:52.97950093 +0000 UTC m=+289.414403136"
Oct 30 16:51:53.024726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:53.024649    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-j2dld"]
Oct 30 16:51:53.028379 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:51:53.028350    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ebb4433_b279_485f_b5c8_7b72cefaec9e.slice/crio-9e9303cc4e45911d8ec34f9d058929485b569c9cfec30cef4a1d893760a0ea4b WatchSource:0}: Error finding container 9e9303cc4e45911d8ec34f9d058929485b569c9cfec30cef4a1d893760a0ea4b: Status 404 returned error can't find the container with id 9e9303cc4e45911d8ec34f9d058929485b569c9cfec30cef4a1d893760a0ea4b
Oct 30 16:51:53.943437 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:53.943384    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-j2dld" event={"ID":"8ebb4433-b279-485f-b5c8-7b72cefaec9e","Type":"ContainerStarted","Data":"9e9303cc4e45911d8ec34f9d058929485b569c9cfec30cef4a1d893760a0ea4b"}
Oct 30 16:51:54.487069 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:54.487035    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"]
Oct 30 16:51:54.490843 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:54.490815    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"
Oct 30 16:51:54.505850 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:54.505818    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 16:51:54.506005 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:54.505858    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 16:51:54.517850 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:54.517807    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-bh5jz\""
Oct 30 16:51:54.520194 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:54.520163    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"]
Oct 30 16:51:54.605919 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:54.605871    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 16:51:54.608248 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:54.608218    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5njlxb\" (UID: \"edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"
Oct 30 16:51:54.608397 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:54.608276    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5njlxb\" (UID: \"edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"
Oct 30 16:51:54.608490 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:54.608395    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5njlxb\" (UID: \"edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"
Oct 30 16:51:54.608556 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:54.608500    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr776\" (UniqueName: \"kubernetes.io/projected/edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a-kube-api-access-cr776\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5njlxb\" (UID: \"edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"
Oct 30 16:51:54.608556 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:54.608541    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5njlxb\" (UID: \"edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"
Oct 30 16:51:54.608673 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:54.608579    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5njlxb\" (UID: \"edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"
Oct 30 16:51:54.608673 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:54.608631    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5njlxb\" (UID: \"edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"
Oct 30 16:51:54.608673 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:54.608660    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5njlxb\" (UID: \"edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"
Oct 30 16:51:54.608826 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:54.608689    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5njlxb\" (UID: \"edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"
Oct 30 16:51:54.709778 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:54.709735    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr776\" (UniqueName: \"kubernetes.io/projected/edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a-kube-api-access-cr776\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5njlxb\" (UID: \"edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"
Oct 30 16:51:54.709966 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:54.709795    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5njlxb\" (UID: \"edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"
Oct 30 16:51:54.709966 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:54.709834    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5njlxb\" (UID: \"edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"
Oct 30 16:51:54.709966 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:54.709878    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5njlxb\" (UID: \"edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"
Oct 30 16:51:54.709966 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:54.709896    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5njlxb\" (UID: \"edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"
Oct 30 16:51:54.709966 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:54.709912    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5njlxb\" (UID: \"edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"
Oct 30 16:51:54.709966 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:54.709954    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5njlxb\" (UID: \"edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"
Oct 30 16:51:54.710264 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:54.710014    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5njlxb\" (UID: \"edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"
Oct 30 16:51:54.710264 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:54.710052    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5njlxb\" (UID: \"edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"
Oct 30 16:51:54.710264 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:54.710114    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5njlxb\" (UID: \"edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"
Oct 30 16:51:54.710542 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:54.710509    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5njlxb\" (UID: \"edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"
Oct 30 16:51:54.710542 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:54.710529    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5njlxb\" (UID: \"edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"
Oct 30 16:51:54.710681 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:54.710600    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5njlxb\" (UID: \"edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"
Oct 30 16:51:54.710891 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:54.710864    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5njlxb\" (UID: \"edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"
Oct 30 16:51:54.712832 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:54.712811    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5njlxb\" (UID: \"edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"
Oct 30 16:51:54.713243 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:54.713217    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5njlxb\" (UID: \"edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"
Oct 30 16:51:54.733505 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:54.733477    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5njlxb\" (UID: \"edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"
Oct 30 16:51:54.733660 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:54.733503    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr776\" (UniqueName: \"kubernetes.io/projected/edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a-kube-api-access-cr776\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5njlxb\" (UID: \"edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"
Oct 30 16:51:54.806315 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:54.806227    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"
Oct 30 16:51:55.135893 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:55.135451    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"]
Oct 30 16:51:55.140292 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:51:55.140262    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedfc7ebb_fa0e_44fe_b442_f50ce4e9aa2a.slice/crio-032f1d6a2fb9792067ff7e6422d7adca14a83858692f1c3963cc4556c21bbbfc WatchSource:0}: Error finding container 032f1d6a2fb9792067ff7e6422d7adca14a83858692f1c3963cc4556c21bbbfc: Status 404 returned error can't find the container with id 032f1d6a2fb9792067ff7e6422d7adca14a83858692f1c3963cc4556c21bbbfc
Oct 30 16:51:55.964842 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:55.953947    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb" event={"ID":"edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a","Type":"ContainerStarted","Data":"032f1d6a2fb9792067ff7e6422d7adca14a83858692f1c3963cc4556c21bbbfc"}
Oct 30 16:51:55.971305 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:55.967386    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-j2dld" event={"ID":"8ebb4433-b279-485f-b5c8-7b72cefaec9e","Type":"ContainerStarted","Data":"d69dcd42f1b7ed7b99a62935cb26e3c1f97938f72d56a91bb47938ee5b39efec"}
Oct 30 16:51:55.971305 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:55.967694    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-observability-584f99c788-j2dld"
Oct 30 16:51:56.012286 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:56.012228    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-observability-584f99c788-j2dld" podStartSLOduration=2.424242598 podStartE2EDuration="5.012212729s" totalImagesPullingTime="2.587970131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:56.011545505 +0000 UTC m=+292.446447692" watchObservedRunningTime="2026-10-30 16:51:56.012212729 +0000 UTC m=+292.447114938"
Oct 30 16:51:56.984172 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:51:56.984134    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 16:51:57.922336 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:57.922308    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 16:51:57.922428 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:57.922377    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 16:51:57.922509 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:57.922431    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 16:51:58.984209 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:58.984169    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb" event={"ID":"edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a","Type":"ContainerStarted","Data":"29fc6e4b4e43928fcecc5e6c561757917a170ddde98e1d42831ea688b0e2f051"}
Oct 30 16:51:59.041578 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:59.041515    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb" podStartSLOduration=2.262432888 podStartE2EDuration="5.041497533s" totalImagesPullingTime="2.779064645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:59.040924002 +0000 UTC m=+295.475826208" watchObservedRunningTime="2026-10-30 16:51:59.041497533 +0000 UTC m=+295.476399751"
Oct 30 16:51:59.807257 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:59.807222    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"
Oct 30 16:51:59.811886 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:59.811863    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"
Oct 30 16:51:59.988416 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:59.988382    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"
Oct 30 16:51:59.989425 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:51:59.989383    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5njlxb"
Oct 30 16:52:03.945807 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:03.945767    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-fc9769c4f-f42wp"
Oct 30 16:52:06.976538 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:06.976501    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-observability-584f99c788-j2dld"
Oct 30 16:52:26.976596 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:52:26.976555    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 16:52:53.545103 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.545066    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-6strf"]
Oct 30 16:52:53.548957 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.548933    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-6strf"
Oct 30 16:52:53.586893 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.586865    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-jfxtk\""
Oct 30 16:52:53.587063 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.587008    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 30 16:52:53.588474 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.588454    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-6strf"]
Oct 30 16:52:53.589659 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.589639    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-jt9wq"]
Oct 30 16:52:53.592998 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.592980    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jt9wq"
Oct 30 16:52:53.610228 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.610201    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-jt9wq"]
Oct 30 16:52:53.622351 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.622324    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-nt54h"]
Oct 30 16:52:53.625691 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.625670    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-nt54h"
Oct 30 16:52:53.649169 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.649140    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-nt54h"]
Oct 30 16:52:53.677346 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.677317    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-zhx25"]
Oct 30 16:52:53.679317 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.679291    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 30 16:52:53.679489 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.679393    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-674k7\""
Oct 30 16:52:53.679602 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.679544    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 30 16:52:53.679663 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.679618    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-hcdk2\""
Oct 30 16:52:53.687746 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.687725    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-zhx25"
Oct 30 16:52:53.703706 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.703683    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-zhx25"]
Oct 30 16:52:53.705150 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.705130    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 30 16:52:53.707166 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.705653    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 30 16:52:53.707166 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.705835    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-2pcqn\""
Oct 30 16:52:53.714360 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.714337    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpzsg\" (UniqueName: \"kubernetes.io/projected/24a380dc-cf10-49d0-a874-db00438648dc-kube-api-access-tpzsg\") pod \"kserve-controller-manager-776cc66d47-6strf\" (UID: \"24a380dc-cf10-49d0-a874-db00438648dc\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6strf"
Oct 30 16:52:53.714473 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.714371    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24a380dc-cf10-49d0-a874-db00438648dc-cert\") pod \"kserve-controller-manager-776cc66d47-6strf\" (UID: \"24a380dc-cf10-49d0-a874-db00438648dc\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6strf"
Oct 30 16:52:53.714473 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.714398    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlg85\" (UniqueName: \"kubernetes.io/projected/bc74354d-23e9-49bf-bc42-82fe6000b4f8-kube-api-access-zlg85\") pod \"llmisvc-controller-manager-869d5dcc5d-jt9wq\" (UID: \"bc74354d-23e9-49bf-bc42-82fe6000b4f8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jt9wq"
Oct 30 16:52:53.714547 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.714530    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/515a8205-3b3b-4e80-9b96-79c33acfdc06-tls-certs\") pod \"model-serving-api-75bd89fdfc-nt54h\" (UID: \"515a8205-3b3b-4e80-9b96-79c33acfdc06\") " pod="opendatahub/model-serving-api-75bd89fdfc-nt54h"
Oct 30 16:52:53.714580 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.714556    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7l2h\" (UniqueName: \"kubernetes.io/projected/515a8205-3b3b-4e80-9b96-79c33acfdc06-kube-api-access-k7l2h\") pod \"model-serving-api-75bd89fdfc-nt54h\" (UID: \"515a8205-3b3b-4e80-9b96-79c33acfdc06\") " pod="opendatahub/model-serving-api-75bd89fdfc-nt54h"
Oct 30 16:52:53.714580 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.714577    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc74354d-23e9-49bf-bc42-82fe6000b4f8-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-jt9wq\" (UID: \"bc74354d-23e9-49bf-bc42-82fe6000b4f8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jt9wq"
Oct 30 16:52:53.815915 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.815846    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpzsg\" (UniqueName: \"kubernetes.io/projected/24a380dc-cf10-49d0-a874-db00438648dc-kube-api-access-tpzsg\") pod \"kserve-controller-manager-776cc66d47-6strf\" (UID: \"24a380dc-cf10-49d0-a874-db00438648dc\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6strf"
Oct 30 16:52:53.815915 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.815883    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24a380dc-cf10-49d0-a874-db00438648dc-cert\") pod \"kserve-controller-manager-776cc66d47-6strf\" (UID: \"24a380dc-cf10-49d0-a874-db00438648dc\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6strf"
Oct 30 16:52:53.816091 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.815914    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfrvs\" (UniqueName: \"kubernetes.io/projected/4452411d-5b03-40ae-a3d3-e96adf966c46-kube-api-access-jfrvs\") pod \"odh-model-controller-f87fcf56c-zhx25\" (UID: \"4452411d-5b03-40ae-a3d3-e96adf966c46\") " pod="opendatahub/odh-model-controller-f87fcf56c-zhx25"
Oct 30 16:52:53.816091 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.816053    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlg85\" (UniqueName: \"kubernetes.io/projected/bc74354d-23e9-49bf-bc42-82fe6000b4f8-kube-api-access-zlg85\") pod \"llmisvc-controller-manager-869d5dcc5d-jt9wq\" (UID: \"bc74354d-23e9-49bf-bc42-82fe6000b4f8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jt9wq"
Oct 30 16:52:53.816091 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.816080    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4452411d-5b03-40ae-a3d3-e96adf966c46-cert\") pod \"odh-model-controller-f87fcf56c-zhx25\" (UID: \"4452411d-5b03-40ae-a3d3-e96adf966c46\") " pod="opendatahub/odh-model-controller-f87fcf56c-zhx25"
Oct 30 16:52:53.816230 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.816111    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4452411d-5b03-40ae-a3d3-e96adf966c46-metrics-certs\") pod \"odh-model-controller-f87fcf56c-zhx25\" (UID: \"4452411d-5b03-40ae-a3d3-e96adf966c46\") " pod="opendatahub/odh-model-controller-f87fcf56c-zhx25"
Oct 30 16:52:53.816230 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.816201    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/515a8205-3b3b-4e80-9b96-79c33acfdc06-tls-certs\") pod \"model-serving-api-75bd89fdfc-nt54h\" (UID: \"515a8205-3b3b-4e80-9b96-79c33acfdc06\") " pod="opendatahub/model-serving-api-75bd89fdfc-nt54h"
Oct 30 16:52:53.816316 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.816281    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7l2h\" (UniqueName: \"kubernetes.io/projected/515a8205-3b3b-4e80-9b96-79c33acfdc06-kube-api-access-k7l2h\") pod \"model-serving-api-75bd89fdfc-nt54h\" (UID: \"515a8205-3b3b-4e80-9b96-79c33acfdc06\") " pod="opendatahub/model-serving-api-75bd89fdfc-nt54h"
Oct 30 16:52:53.816365 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.816314    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc74354d-23e9-49bf-bc42-82fe6000b4f8-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-jt9wq\" (UID: \"bc74354d-23e9-49bf-bc42-82fe6000b4f8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jt9wq"
Oct 30 16:52:53.818539 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.818515    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24a380dc-cf10-49d0-a874-db00438648dc-cert\") pod \"kserve-controller-manager-776cc66d47-6strf\" (UID: \"24a380dc-cf10-49d0-a874-db00438648dc\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6strf"
Oct 30 16:52:53.818673 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.818583    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/515a8205-3b3b-4e80-9b96-79c33acfdc06-tls-certs\") pod \"model-serving-api-75bd89fdfc-nt54h\" (UID: \"515a8205-3b3b-4e80-9b96-79c33acfdc06\") " pod="opendatahub/model-serving-api-75bd89fdfc-nt54h"
Oct 30 16:52:53.818864 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.818843    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc74354d-23e9-49bf-bc42-82fe6000b4f8-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-jt9wq\" (UID: \"bc74354d-23e9-49bf-bc42-82fe6000b4f8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jt9wq"
Oct 30 16:52:53.840688 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.840667    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlg85\" (UniqueName: \"kubernetes.io/projected/bc74354d-23e9-49bf-bc42-82fe6000b4f8-kube-api-access-zlg85\") pod \"llmisvc-controller-manager-869d5dcc5d-jt9wq\" (UID: \"bc74354d-23e9-49bf-bc42-82fe6000b4f8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jt9wq"
Oct 30 16:52:53.840792 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.840724    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7l2h\" (UniqueName: \"kubernetes.io/projected/515a8205-3b3b-4e80-9b96-79c33acfdc06-kube-api-access-k7l2h\") pod \"model-serving-api-75bd89fdfc-nt54h\" (UID: \"515a8205-3b3b-4e80-9b96-79c33acfdc06\") " pod="opendatahub/model-serving-api-75bd89fdfc-nt54h"
Oct 30 16:52:53.840792 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.840765    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpzsg\" (UniqueName: \"kubernetes.io/projected/24a380dc-cf10-49d0-a874-db00438648dc-kube-api-access-tpzsg\") pod \"kserve-controller-manager-776cc66d47-6strf\" (UID: \"24a380dc-cf10-49d0-a874-db00438648dc\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6strf"
Oct 30 16:52:53.861153 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.861130    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-6strf"
Oct 30 16:52:53.904994 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.904961    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jt9wq"
Oct 30 16:52:53.921230 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.917393    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfrvs\" (UniqueName: \"kubernetes.io/projected/4452411d-5b03-40ae-a3d3-e96adf966c46-kube-api-access-jfrvs\") pod \"odh-model-controller-f87fcf56c-zhx25\" (UID: \"4452411d-5b03-40ae-a3d3-e96adf966c46\") " pod="opendatahub/odh-model-controller-f87fcf56c-zhx25"
Oct 30 16:52:53.921230 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.917489    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4452411d-5b03-40ae-a3d3-e96adf966c46-cert\") pod \"odh-model-controller-f87fcf56c-zhx25\" (UID: \"4452411d-5b03-40ae-a3d3-e96adf966c46\") " pod="opendatahub/odh-model-controller-f87fcf56c-zhx25"
Oct 30 16:52:53.921230 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.917528    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4452411d-5b03-40ae-a3d3-e96adf966c46-metrics-certs\") pod \"odh-model-controller-f87fcf56c-zhx25\" (UID: \"4452411d-5b03-40ae-a3d3-e96adf966c46\") " pod="opendatahub/odh-model-controller-f87fcf56c-zhx25"
Oct 30 16:52:53.921230 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.920010    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4452411d-5b03-40ae-a3d3-e96adf966c46-metrics-certs\") pod \"odh-model-controller-f87fcf56c-zhx25\" (UID: \"4452411d-5b03-40ae-a3d3-e96adf966c46\") " pod="opendatahub/odh-model-controller-f87fcf56c-zhx25"
Oct 30 16:52:53.923712 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.923681    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4452411d-5b03-40ae-a3d3-e96adf966c46-cert\") pod \"odh-model-controller-f87fcf56c-zhx25\" (UID: \"4452411d-5b03-40ae-a3d3-e96adf966c46\") " pod="opendatahub/odh-model-controller-f87fcf56c-zhx25"
Oct 30 16:52:53.937888 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.937618    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-nt54h"
Oct 30 16:52:53.941764 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:53.941743    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfrvs\" (UniqueName: \"kubernetes.io/projected/4452411d-5b03-40ae-a3d3-e96adf966c46-kube-api-access-jfrvs\") pod \"odh-model-controller-f87fcf56c-zhx25\" (UID: \"4452411d-5b03-40ae-a3d3-e96adf966c46\") " pod="opendatahub/odh-model-controller-f87fcf56c-zhx25"
Oct 30 16:52:54.002416 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:54.002106    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-zhx25"
Oct 30 16:52:54.158592 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:54.158521    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-6strf"]
Oct 30 16:52:54.162919 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:52:54.162877    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a380dc_cf10_49d0_a874_db00438648dc.slice/crio-ee90eb8db6aca6cb3ac5edc792a9bbe8cb089166118dcb6ff5c70fbbd1116ba3 WatchSource:0}: Error finding container ee90eb8db6aca6cb3ac5edc792a9bbe8cb089166118dcb6ff5c70fbbd1116ba3: Status 404 returned error can't find the container with id ee90eb8db6aca6cb3ac5edc792a9bbe8cb089166118dcb6ff5c70fbbd1116ba3
Oct 30 16:52:54.164620 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:54.164601    4433 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 16:52:54.208196 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:54.208099    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-jt9wq"]
Oct 30 16:52:54.209627 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:54.209559    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-6strf" event={"ID":"24a380dc-cf10-49d0-a874-db00438648dc","Type":"ContainerStarted","Data":"ee90eb8db6aca6cb3ac5edc792a9bbe8cb089166118dcb6ff5c70fbbd1116ba3"}
Oct 30 16:52:54.211695 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:52:54.211672    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbc74354d_23e9_49bf_bc42_82fe6000b4f8.slice/crio-83ba0dfd9fd46db804c910712eda1d04bf6400eda94bb367e064b7bef421c41e WatchSource:0}: Error finding container 83ba0dfd9fd46db804c910712eda1d04bf6400eda94bb367e064b7bef421c41e: Status 404 returned error can't find the container with id 83ba0dfd9fd46db804c910712eda1d04bf6400eda94bb367e064b7bef421c41e
Oct 30 16:52:54.243832 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:54.243792    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-nt54h"]
Oct 30 16:52:54.247508 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:52:54.247481    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod515a8205_3b3b_4e80_9b96_79c33acfdc06.slice/crio-701c32e39f08d04fb69060aca5723526a3401f4160cef92632bbee20ba0a0ad9 WatchSource:0}: Error finding container 701c32e39f08d04fb69060aca5723526a3401f4160cef92632bbee20ba0a0ad9: Status 404 returned error can't find the container with id 701c32e39f08d04fb69060aca5723526a3401f4160cef92632bbee20ba0a0ad9
Oct 30 16:52:54.301632 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:54.301593    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-zhx25"]
Oct 30 16:52:54.305570 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:52:54.305544    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4452411d_5b03_40ae_a3d3_e96adf966c46.slice/crio-ffdbdaadc42d0749f3f28ff22c8bb4dbefbac677dcc024d89fa7569dc4645734 WatchSource:0}: Error finding container ffdbdaadc42d0749f3f28ff22c8bb4dbefbac677dcc024d89fa7569dc4645734: Status 404 returned error can't find the container with id ffdbdaadc42d0749f3f28ff22c8bb4dbefbac677dcc024d89fa7569dc4645734
Oct 30 16:52:55.229362 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:55.229322    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-zhx25" event={"ID":"4452411d-5b03-40ae-a3d3-e96adf966c46","Type":"ContainerStarted","Data":"ffdbdaadc42d0749f3f28ff22c8bb4dbefbac677dcc024d89fa7569dc4645734"}
Oct 30 16:52:55.231765 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:55.231648    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-nt54h" event={"ID":"515a8205-3b3b-4e80-9b96-79c33acfdc06","Type":"ContainerStarted","Data":"701c32e39f08d04fb69060aca5723526a3401f4160cef92632bbee20ba0a0ad9"}
Oct 30 16:52:55.236113 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:55.236043    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jt9wq" event={"ID":"bc74354d-23e9-49bf-bc42-82fe6000b4f8","Type":"ContainerStarted","Data":"83ba0dfd9fd46db804c910712eda1d04bf6400eda94bb367e064b7bef421c41e"}
Oct 30 16:52:56.996880 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:52:56.996842    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 16:52:58.254224 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:58.254179    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-nt54h" event={"ID":"515a8205-3b3b-4e80-9b96-79c33acfdc06","Type":"ContainerStarted","Data":"a5c4d1cabe74d0c5c7465f3bdac9e2a47f2b10004229d64793dda061d0c094f4"}
Oct 30 16:52:58.254726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:58.254369    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-nt54h"
Oct 30 16:52:58.296671 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:58.296623    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-nt54h" podStartSLOduration=2.223249892 podStartE2EDuration="5.296608824s" totalImagesPullingTime="3.073358932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:58.295829338 +0000 UTC m=+354.730731571" watchObservedRunningTime="2026-10-30 16:52:58.296608824 +0000 UTC m=+354.731511030"
Oct 30 16:52:59.260872 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:59.260826    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-zhx25" event={"ID":"4452411d-5b03-40ae-a3d3-e96adf966c46","Type":"ContainerStarted","Data":"f79026030eca1f9d6c914d240e90a1af77b034e8e03d94512824729127f1e993"}
Oct 30 16:52:59.261395 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:59.260968    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-f87fcf56c-zhx25"
Oct 30 16:52:59.262568 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:59.262539    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jt9wq" event={"ID":"bc74354d-23e9-49bf-bc42-82fe6000b4f8","Type":"ContainerStarted","Data":"856a13d5eb262cfe1baf337f38c90fbea9df2e6d809ce4d3a0929b438d7c006b"}
Oct 30 16:52:59.262693 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:59.262590    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jt9wq"
Oct 30 16:52:59.264124 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:59.264100    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-6strf" event={"ID":"24a380dc-cf10-49d0-a874-db00438648dc","Type":"ContainerStarted","Data":"daf6c88a63cf293af43ad433937a8cf6587262893c4b47ceca185b4ee7b8f3c1"}
Oct 30 16:52:59.264236 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:59.264139    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-6strf"
Oct 30 16:52:59.302460 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:59.302376    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-f87fcf56c-zhx25" podStartSLOduration=2.136258691 podStartE2EDuration="6.302352024s" totalImagesPullingTime="4.166093333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:59.301750637 +0000 UTC m=+355.736652833" watchObservedRunningTime="2026-10-30 16:52:59.302352024 +0000 UTC m=+355.737254232"
Oct 30 16:52:59.342155 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:59.342108    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jt9wq" podStartSLOduration=2.274485354 podStartE2EDuration="6.342095666s" totalImagesPullingTime="4.067610312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:59.341531103 +0000 UTC m=+355.776433336" watchObservedRunningTime="2026-10-30 16:52:59.342095666 +0000 UTC m=+355.776997868"
Oct 30 16:52:59.383392 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:52:59.383338    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-6strf" podStartSLOduration=2.227966433 podStartE2EDuration="6.383319671s" totalImagesPullingTime="4.155353238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:59.38136358 +0000 UTC m=+355.816265783" watchObservedRunningTime="2026-10-30 16:52:59.383319671 +0000 UTC m=+355.818221869"
Oct 30 16:53:09.268123 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:53:09.268094    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-nt54h"
Oct 30 16:53:10.271773 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:53:10.271743    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-f87fcf56c-zhx25"
Oct 30 16:53:26.976956 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:53:26.976921    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 16:53:30.271711 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:53:30.271680    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jt9wq"
Oct 30 16:53:30.274757 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:53:30.274738    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-6strf"
Oct 30 16:53:56.975668 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:53:56.975580    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 16:54:01.663569 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:54:01.663531    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-428zq"]
Oct 30 16:54:01.667239 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:54:01.667219    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-428zq"
Oct 30 16:54:01.693986 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:54:01.693957    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-428zq"]
Oct 30 16:54:01.700090 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:54:01.700057    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 16:54:01.700090 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:54:01.700068    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 16:54:01.700238 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:54:01.700118    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-7c552\""
Oct 30 16:54:01.799425 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:54:01.799378    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndjfc\" (UniqueName: \"kubernetes.io/projected/7c756c06-62e5-449c-8ed7-d17d45c33839-kube-api-access-ndjfc\") pod \"authorino-operator-68d6dd76dc-428zq\" (UID: \"7c756c06-62e5-449c-8ed7-d17d45c33839\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-428zq"
Oct 30 16:54:01.900296 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:54:01.900257    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndjfc\" (UniqueName: \"kubernetes.io/projected/7c756c06-62e5-449c-8ed7-d17d45c33839-kube-api-access-ndjfc\") pod \"authorino-operator-68d6dd76dc-428zq\" (UID: \"7c756c06-62e5-449c-8ed7-d17d45c33839\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-428zq"
Oct 30 16:54:01.927970 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:54:01.927943    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndjfc\" (UniqueName: \"kubernetes.io/projected/7c756c06-62e5-449c-8ed7-d17d45c33839-kube-api-access-ndjfc\") pod \"authorino-operator-68d6dd76dc-428zq\" (UID: \"7c756c06-62e5-449c-8ed7-d17d45c33839\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-428zq"
Oct 30 16:54:01.979811 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:54:01.979770    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-428zq"
Oct 30 16:54:02.328426 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:54:02.328207    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-428zq"]
Oct 30 16:54:02.331383 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:54:02.331349    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c756c06_62e5_449c_8ed7_d17d45c33839.slice/crio-dbd1e5d573bff490fb0a15d5cca7c7aae176966b60d9a0f9ed975a7b56f7e255 WatchSource:0}: Error finding container dbd1e5d573bff490fb0a15d5cca7c7aae176966b60d9a0f9ed975a7b56f7e255: Status 404 returned error can't find the container with id dbd1e5d573bff490fb0a15d5cca7c7aae176966b60d9a0f9ed975a7b56f7e255
Oct 30 16:54:02.530996 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:54:02.530960    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-428zq" event={"ID":"7c756c06-62e5-449c-8ed7-d17d45c33839","Type":"ContainerStarted","Data":"dbd1e5d573bff490fb0a15d5cca7c7aae176966b60d9a0f9ed975a7b56f7e255"}
Oct 30 16:54:05.544858 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:54:05.544815    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-428zq" event={"ID":"7c756c06-62e5-449c-8ed7-d17d45c33839","Type":"ContainerStarted","Data":"86b6da2151e46f18e5c79de2d9a427408912b06ffe1061c96028fbe1108ded19"}
Oct 30 16:54:05.545334 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:54:05.544929    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-428zq"
Oct 30 16:54:05.589332 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:54:05.589268    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d6dd76dc-428zq" podStartSLOduration=2.373101644 podStartE2EDuration="4.589253888s" totalImagesPullingTime="2.216152244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:54:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:54:05.588136932 +0000 UTC m=+422.023039139" watchObservedRunningTime="2026-10-30 16:54:05.589253888 +0000 UTC m=+422.024156103"
Oct 30 16:54:16.550674 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:54:16.550643    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-428zq"
Oct 30 16:54:26.977052 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:54:26.977007    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 16:54:56.975887 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:54:56.975846    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 16:55:03.294208 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:55:03.294176    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 16:55:03.297634 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:55:03.297613    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"
Oct 30 16:55:03.315927 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:55:03.315898    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 16:55:03.324250 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:55:03.324225    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wt7tg\""
Oct 30 16:55:03.326002 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:55:03.325979    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 16:55:03.387505 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:55:03.387448    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 16:55:03.438349 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:55:03.438317    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m25p2\" (UniqueName: \"kubernetes.io/projected/f6e2dab6-5a9d-49a1-ad32-1680c0d2af56-kube-api-access-m25p2\") pod \"limitador-limitador-7c9b87d574-mbzbg\" (UID: \"f6e2dab6-5a9d-49a1-ad32-1680c0d2af56\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"
Oct 30 16:55:03.438519 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:55:03.438482    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f6e2dab6-5a9d-49a1-ad32-1680c0d2af56-config-file\") pod \"limitador-limitador-7c9b87d574-mbzbg\" (UID: \"f6e2dab6-5a9d-49a1-ad32-1680c0d2af56\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"
Oct 30 16:55:03.538992 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:55:03.538940    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m25p2\" (UniqueName: \"kubernetes.io/projected/f6e2dab6-5a9d-49a1-ad32-1680c0d2af56-kube-api-access-m25p2\") pod \"limitador-limitador-7c9b87d574-mbzbg\" (UID: \"f6e2dab6-5a9d-49a1-ad32-1680c0d2af56\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"
Oct 30 16:55:03.539173 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:55:03.539058    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f6e2dab6-5a9d-49a1-ad32-1680c0d2af56-config-file\") pod \"limitador-limitador-7c9b87d574-mbzbg\" (UID: \"f6e2dab6-5a9d-49a1-ad32-1680c0d2af56\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"
Oct 30 16:55:03.539667 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:55:03.539650    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f6e2dab6-5a9d-49a1-ad32-1680c0d2af56-config-file\") pod \"limitador-limitador-7c9b87d574-mbzbg\" (UID: \"f6e2dab6-5a9d-49a1-ad32-1680c0d2af56\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"
Oct 30 16:55:03.562920 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:55:03.562865    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m25p2\" (UniqueName: \"kubernetes.io/projected/f6e2dab6-5a9d-49a1-ad32-1680c0d2af56-kube-api-access-m25p2\") pod \"limitador-limitador-7c9b87d574-mbzbg\" (UID: \"f6e2dab6-5a9d-49a1-ad32-1680c0d2af56\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"
Oct 30 16:55:03.612616 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:55:03.612582    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"
Oct 30 16:55:03.909477 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:55:03.909388    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 16:55:03.913375 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:55:03.913350    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e2dab6_5a9d_49a1_ad32_1680c0d2af56.slice/crio-b9982b219f055bba8512c497555f76333486a40bf0f0ffa8bbaadf0230c8fb8c WatchSource:0}: Error finding container b9982b219f055bba8512c497555f76333486a40bf0f0ffa8bbaadf0230c8fb8c: Status 404 returned error can't find the container with id b9982b219f055bba8512c497555f76333486a40bf0f0ffa8bbaadf0230c8fb8c
Oct 30 16:55:04.785022 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:55:04.784988    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-mbzbg" event={"ID":"f6e2dab6-5a9d-49a1-ad32-1680c0d2af56","Type":"ContainerStarted","Data":"b9982b219f055bba8512c497555f76333486a40bf0f0ffa8bbaadf0230c8fb8c"}
Oct 30 16:55:05.790484 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:55:05.790446    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-mbzbg" event={"ID":"f6e2dab6-5a9d-49a1-ad32-1680c0d2af56","Type":"ContainerStarted","Data":"aba7d14de1fcc511c382b48dca79a1d0b85875ad2ffc49e438137eb7fab1da1d"}
Oct 30 16:55:05.790896 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:55:05.790513    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"
Oct 30 16:55:05.831805 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:55:05.831741    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-mbzbg" podStartSLOduration=1.836804536 podStartE2EDuration="2.831725305s" totalImagesPullingTime="994.920769ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:55:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:55:05.830664963 +0000 UTC m=+482.265567170" watchObservedRunningTime="2026-10-30 16:55:05.831725305 +0000 UTC m=+482.266627511"
Oct 30 16:55:16.796396 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:55:16.796325    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"
Oct 30 16:55:26.976101 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:55:26.976065    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 16:55:56.976289 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:55:56.976250    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 16:56:20.174932 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:56:20.174886    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 16:56:26.976071 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:56:26.976033    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 16:56:28.349984 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:56:28.349949    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-54btj"]
Oct 30 16:56:28.353279 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:56:28.353263    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-54btj"
Oct 30 16:56:28.380054 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:56:28.380020    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-54btj"]
Oct 30 16:56:28.451350 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:56:28.451309    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93b3e2e5-f146-47a4-92b3-189bfdb8b080-cert\") pod \"llmisvc-controller-manager-6466c554d8-54btj\" (UID: \"93b3e2e5-f146-47a4-92b3-189bfdb8b080\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-54btj"
Oct 30 16:56:28.451552 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:56:28.451358    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2jxg\" (UniqueName: \"kubernetes.io/projected/93b3e2e5-f146-47a4-92b3-189bfdb8b080-kube-api-access-c2jxg\") pod \"llmisvc-controller-manager-6466c554d8-54btj\" (UID: \"93b3e2e5-f146-47a4-92b3-189bfdb8b080\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-54btj"
Oct 30 16:56:28.552753 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:56:28.552716    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93b3e2e5-f146-47a4-92b3-189bfdb8b080-cert\") pod \"llmisvc-controller-manager-6466c554d8-54btj\" (UID: \"93b3e2e5-f146-47a4-92b3-189bfdb8b080\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-54btj"
Oct 30 16:56:28.552915 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:56:28.552765    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2jxg\" (UniqueName: \"kubernetes.io/projected/93b3e2e5-f146-47a4-92b3-189bfdb8b080-kube-api-access-c2jxg\") pod \"llmisvc-controller-manager-6466c554d8-54btj\" (UID: \"93b3e2e5-f146-47a4-92b3-189bfdb8b080\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-54btj"
Oct 30 16:56:28.555147 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:56:28.555125    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93b3e2e5-f146-47a4-92b3-189bfdb8b080-cert\") pod \"llmisvc-controller-manager-6466c554d8-54btj\" (UID: \"93b3e2e5-f146-47a4-92b3-189bfdb8b080\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-54btj"
Oct 30 16:56:28.577941 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:56:28.577902    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2jxg\" (UniqueName: \"kubernetes.io/projected/93b3e2e5-f146-47a4-92b3-189bfdb8b080-kube-api-access-c2jxg\") pod \"llmisvc-controller-manager-6466c554d8-54btj\" (UID: \"93b3e2e5-f146-47a4-92b3-189bfdb8b080\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-54btj"
Oct 30 16:56:28.665839 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:56:28.665759    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-54btj"
Oct 30 16:56:28.962320 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:56:28.962288    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-54btj"]
Oct 30 16:56:28.965827 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:56:28.965790    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b3e2e5_f146_47a4_92b3_189bfdb8b080.slice/crio-10d518964e716196c5e51b674847754fbe8b1dbe7631d744430fceda178f9895 WatchSource:0}: Error finding container 10d518964e716196c5e51b674847754fbe8b1dbe7631d744430fceda178f9895: Status 404 returned error can't find the container with id 10d518964e716196c5e51b674847754fbe8b1dbe7631d744430fceda178f9895
Oct 30 16:56:29.141678 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:56:29.141645    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-54btj" event={"ID":"93b3e2e5-f146-47a4-92b3-189bfdb8b080","Type":"ContainerStarted","Data":"10d518964e716196c5e51b674847754fbe8b1dbe7631d744430fceda178f9895"}
Oct 30 16:56:30.147256 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:56:30.147212    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-54btj" event={"ID":"93b3e2e5-f146-47a4-92b3-189bfdb8b080","Type":"ContainerStarted","Data":"bebbbb7194ee571c1b3e6dede1ab37294955db94dad2d63af2ba604aa4176f4b"}
Oct 30 16:56:30.147766 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:56:30.147390    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-54btj"
Oct 30 16:56:30.187783 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:56:30.187724    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-54btj" podStartSLOduration=1.8439738540000001 podStartE2EDuration="2.187709663s" totalImagesPullingTime="343.735809ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:56:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:56:30.187307368 +0000 UTC m=+566.622209575" watchObservedRunningTime="2026-10-30 16:56:30.187709663 +0000 UTC m=+566.622611868"
Oct 30 16:56:56.976923 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:56:56.976844    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 16:57:01.153087 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:01.153057    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-54btj"
Oct 30 16:57:01.222628 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:01.222593    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-jt9wq"]
Oct 30 16:57:01.222876 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:01.222838    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jt9wq" podUID="bc74354d-23e9-49bf-bc42-82fe6000b4f8" containerName="manager" containerID="cri-o://856a13d5eb262cfe1baf337f38c90fbea9df2e6d809ce4d3a0929b438d7c006b" gracePeriod=30
Oct 30 16:57:01.465125 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:01.465097    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jt9wq"
Oct 30 16:57:01.551970 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:01.551943    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc74354d-23e9-49bf-bc42-82fe6000b4f8-kube-api-access-zlg85\" (UniqueName: \"kubernetes.io/projected/bc74354d-23e9-49bf-bc42-82fe6000b4f8-kube-api-access-zlg85\") pod \"bc74354d-23e9-49bf-bc42-82fe6000b4f8\" (UID: \"bc74354d-23e9-49bf-bc42-82fe6000b4f8\") "
Oct 30 16:57:01.552117 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:01.551978    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc74354d-23e9-49bf-bc42-82fe6000b4f8-cert\" (UniqueName: \"kubernetes.io/secret/bc74354d-23e9-49bf-bc42-82fe6000b4f8-cert\") pod \"bc74354d-23e9-49bf-bc42-82fe6000b4f8\" (UID: \"bc74354d-23e9-49bf-bc42-82fe6000b4f8\") "
Oct 30 16:57:01.554034 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:01.554013    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc74354d-23e9-49bf-bc42-82fe6000b4f8-cert" pod "bc74354d-23e9-49bf-bc42-82fe6000b4f8" (UID: "bc74354d-23e9-49bf-bc42-82fe6000b4f8"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:57:01.554161 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:01.554140    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc74354d-23e9-49bf-bc42-82fe6000b4f8-kube-api-access-zlg85" pod "bc74354d-23e9-49bf-bc42-82fe6000b4f8" (UID: "bc74354d-23e9-49bf-bc42-82fe6000b4f8"). InnerVolumeSpecName "kube-api-access-zlg85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:57:01.652951 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:01.652917    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlg85\" (UniqueName: \"kubernetes.io/projected/bc74354d-23e9-49bf-bc42-82fe6000b4f8-kube-api-access-zlg85\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:57:01.652951 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:01.652946    4433 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc74354d-23e9-49bf-bc42-82fe6000b4f8-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 16:57:02.287373 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:02.287341    4433 generic.go:356] "Generic (PLEG): container finished" podID="bc74354d-23e9-49bf-bc42-82fe6000b4f8" containerID="856a13d5eb262cfe1baf337f38c90fbea9df2e6d809ce4d3a0929b438d7c006b" exitCode=0
Oct 30 16:57:02.287894 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:02.287398    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jt9wq"
Oct 30 16:57:02.287894 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:02.287428    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jt9wq" event={"ID":"bc74354d-23e9-49bf-bc42-82fe6000b4f8","Type":"ContainerDied","Data":"856a13d5eb262cfe1baf337f38c90fbea9df2e6d809ce4d3a0929b438d7c006b"}
Oct 30 16:57:02.287894 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:02.287468    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jt9wq" event={"ID":"bc74354d-23e9-49bf-bc42-82fe6000b4f8","Type":"ContainerDied","Data":"83ba0dfd9fd46db804c910712eda1d04bf6400eda94bb367e064b7bef421c41e"}
Oct 30 16:57:02.287894 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:02.287490    4433 scope.go:122] "RemoveContainer" containerID="856a13d5eb262cfe1baf337f38c90fbea9df2e6d809ce4d3a0929b438d7c006b"
Oct 30 16:57:02.295706 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:02.295688    4433 scope.go:122] "RemoveContainer" containerID="856a13d5eb262cfe1baf337f38c90fbea9df2e6d809ce4d3a0929b438d7c006b"
Oct 30 16:57:02.295977 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:57:02.295958    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"856a13d5eb262cfe1baf337f38c90fbea9df2e6d809ce4d3a0929b438d7c006b\": container with ID starting with 856a13d5eb262cfe1baf337f38c90fbea9df2e6d809ce4d3a0929b438d7c006b not found: ID does not exist" containerID="856a13d5eb262cfe1baf337f38c90fbea9df2e6d809ce4d3a0929b438d7c006b"
Oct 30 16:57:02.296034 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:02.295987    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"856a13d5eb262cfe1baf337f38c90fbea9df2e6d809ce4d3a0929b438d7c006b"} err="failed to get container status \"856a13d5eb262cfe1baf337f38c90fbea9df2e6d809ce4d3a0929b438d7c006b\": rpc error: code = NotFound desc = could not find container \"856a13d5eb262cfe1baf337f38c90fbea9df2e6d809ce4d3a0929b438d7c006b\": container with ID starting with 856a13d5eb262cfe1baf337f38c90fbea9df2e6d809ce4d3a0929b438d7c006b not found: ID does not exist"
Oct 30 16:57:02.377483 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:02.377452    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-jt9wq"]
Oct 30 16:57:02.420192 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:02.420157    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-jt9wq"]
Oct 30 16:57:03.694104 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:03.694074    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc74354d-23e9-49bf-bc42-82fe6000b4f8" path="/var/lib/kubelet/pods/bc74354d-23e9-49bf-bc42-82fe6000b4f8/volumes"
Oct 30 16:57:21.152534 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:21.152502    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 16:57:24.510730 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:24.510693    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 16:57:26.977696 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:57:26.977654    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 16:57:29.097644 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:29.097611    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 16:57:34.446381 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:34.446336    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 16:57:40.442985 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:40.442946    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 16:57:48.112035 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.111999    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"]
Oct 30 16:57:48.112589 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.112569    4433 cpu_manager.go:414] "containerMap: removing container" podUID="bc74354d-23e9-49bf-bc42-82fe6000b4f8" containerName="manager"
Oct 30 16:57:48.112663 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.112593    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc74354d-23e9-49bf-bc42-82fe6000b4f8" containerName="manager"
Oct 30 16:57:48.112726 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.112713    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc74354d-23e9-49bf-bc42-82fe6000b4f8" containerName="manager"
Oct 30 16:57:48.115967 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.115945    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"
Oct 30 16:57:48.133172 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.133145    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fcls9\""
Oct 30 16:57:48.146635 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.146603    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"]
Oct 30 16:57:48.163359 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.163325    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8vzdk\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"
Oct 30 16:57:48.164417 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.164364    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pxhl\" (UniqueName: \"kubernetes.io/projected/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-kube-api-access-8pxhl\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8vzdk\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"
Oct 30 16:57:48.164528 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.164497    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8vzdk\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"
Oct 30 16:57:48.164585 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.164556    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8vzdk\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"
Oct 30 16:57:48.164643 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.164581    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8vzdk\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"
Oct 30 16:57:48.164691 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.164645    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8vzdk\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"
Oct 30 16:57:48.164735 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.164689    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8vzdk\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"
Oct 30 16:57:48.164735 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.164723    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8vzdk\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"
Oct 30 16:57:48.164809 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.164751    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8vzdk\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"
Oct 30 16:57:48.265676 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.265634    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8vzdk\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"
Oct 30 16:57:48.265676 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.265686    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8vzdk\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"
Oct 30 16:57:48.265900 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.265755    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8vzdk\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"
Oct 30 16:57:48.265900 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.265784    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pxhl\" (UniqueName: \"kubernetes.io/projected/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-kube-api-access-8pxhl\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8vzdk\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"
Oct 30 16:57:48.265983 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.265903    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8vzdk\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"
Oct 30 16:57:48.265983 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.265958    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8vzdk\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"
Oct 30 16:57:48.266079 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.265983    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8vzdk\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"
Oct 30 16:57:48.266079 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.266029    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8vzdk\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"
Oct 30 16:57:48.266183 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.266076    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8vzdk\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"
Oct 30 16:57:48.266239 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.266184    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8vzdk\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"
Oct 30 16:57:48.266293 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.266246    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8vzdk\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"
Oct 30 16:57:48.266439 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.266417    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8vzdk\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"
Oct 30 16:57:48.266572 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.266550    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8vzdk\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"
Oct 30 16:57:48.266678 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.266658    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8vzdk\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"
Oct 30 16:57:48.268278 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.268245    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8vzdk\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"
Oct 30 16:57:48.268278 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.268249    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8vzdk\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"
Oct 30 16:57:48.288782 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.288751    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8vzdk\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"
Oct 30 16:57:48.288922 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.288871    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pxhl\" (UniqueName: \"kubernetes.io/projected/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-kube-api-access-8pxhl\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8vzdk\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"
Oct 30 16:57:48.429198 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.429168    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"
Oct 30 16:57:48.743482 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.743389    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"]
Oct 30 16:57:48.747111 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:57:48.747085    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72cf77f0_bc7d_4bc1_a655_f4b98a34f3cc.slice/crio-f922764e83698445a3a662630edc4a01d74fdd9c39e5e42966f120a7eea59c63 WatchSource:0}: Error finding container f922764e83698445a3a662630edc4a01d74fdd9c39e5e42966f120a7eea59c63: Status 404 returned error can't find the container with id f922764e83698445a3a662630edc4a01d74fdd9c39e5e42966f120a7eea59c63
Oct 30 16:57:48.749129 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.749100    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 16:57:48.749211 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.749162    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 16:57:48.749211 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:48.749191    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 16:57:49.498494 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:49.498459    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk" event={"ID":"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc","Type":"ContainerStarted","Data":"59c124e9845acc942314f6848c269ad97b968f3f02757fef8ac95366dd3e6220"}
Oct 30 16:57:49.498860 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:49.498513    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk" event={"ID":"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc","Type":"ContainerStarted","Data":"f922764e83698445a3a662630edc4a01d74fdd9c39e5e42966f120a7eea59c63"}
Oct 30 16:57:49.543706 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:49.543653    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk" podStartSLOduration=1.5436407760000002 podStartE2EDuration="1.543640776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:57:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:57:49.542874411 +0000 UTC m=+645.977776622" watchObservedRunningTime="2026-10-30 16:57:49.543640776 +0000 UTC m=+645.978542981"
Oct 30 16:57:50.429393 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:50.429354    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"
Oct 30 16:57:50.430688 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:50.430664    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8vzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 16:57:50.430793 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:50.430730    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk" podUID="72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 16:57:51.430349 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:51.430312    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8vzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 16:57:51.430788 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:51.430361    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk" podUID="72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 16:57:52.430822 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:52.430783    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8vzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 16:57:52.431778 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:52.431375    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk" podUID="72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 16:57:53.430163 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:53.430133    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8vzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 16:57:53.430337 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:53.430187    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk" podUID="72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 16:57:54.430390 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:54.430358    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8vzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 16:57:54.430797 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:54.430418    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk" podUID="72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 16:57:55.430360 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:55.430323    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8vzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 16:57:55.430546 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:55.430373    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk" podUID="72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 16:57:56.925824 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:56.925783    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"
Oct 30 16:57:56.926341 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:56.926289    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"
Oct 30 16:57:56.927078 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:57:56.927049    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"
Oct 30 16:57:56.975377 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:57:56.975340    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 16:58:03.129865 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:03.129831    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"]
Oct 30 16:58:03.133609 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:03.133585    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"
Oct 30 16:58:03.162872 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:03.162845    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"]
Oct 30 16:58:03.207124 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:03.207088    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f64def0-c36b-4a21-be70-77604220eaf1-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-57s2t\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"
Oct 30 16:58:03.207295 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:03.207134    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f64def0-c36b-4a21-be70-77604220eaf1-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-57s2t\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"
Oct 30 16:58:03.207295 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:03.207158    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f64def0-c36b-4a21-be70-77604220eaf1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-57s2t\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"
Oct 30 16:58:03.207295 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:03.207271    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f64def0-c36b-4a21-be70-77604220eaf1-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-57s2t\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"
Oct 30 16:58:03.207431 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:03.207300    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f64def0-c36b-4a21-be70-77604220eaf1-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-57s2t\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"
Oct 30 16:58:03.207431 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:03.207371    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f64def0-c36b-4a21-be70-77604220eaf1-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-57s2t\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"
Oct 30 16:58:03.207507 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:03.207443    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f64def0-c36b-4a21-be70-77604220eaf1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-57s2t\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"
Oct 30 16:58:03.207507 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:03.207474    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f64def0-c36b-4a21-be70-77604220eaf1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-57s2t\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"
Oct 30 16:58:03.207507 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:03.207491    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q58h5\" (UniqueName: \"kubernetes.io/projected/1f64def0-c36b-4a21-be70-77604220eaf1-kube-api-access-q58h5\") pod \"maas-default-gateway-openshift-default-6db586dd6c-57s2t\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"
Oct 30 16:58:03.308216 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:03.308181    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f64def0-c36b-4a21-be70-77604220eaf1-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-57s2t\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"
Oct 30 16:58:03.308216 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:03.308222    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f64def0-c36b-4a21-be70-77604220eaf1-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-57s2t\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"
Oct 30 16:58:03.308539 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:03.308247    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f64def0-c36b-4a21-be70-77604220eaf1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-57s2t\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"
Oct 30 16:58:03.308539 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:03.308263    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f64def0-c36b-4a21-be70-77604220eaf1-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-57s2t\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"
Oct 30 16:58:03.308539 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:03.308285    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f64def0-c36b-4a21-be70-77604220eaf1-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-57s2t\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"
Oct 30 16:58:03.308539 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:03.308323    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f64def0-c36b-4a21-be70-77604220eaf1-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-57s2t\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"
Oct 30 16:58:03.308539 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:03.308455    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f64def0-c36b-4a21-be70-77604220eaf1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-57s2t\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"
Oct 30 16:58:03.308539 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:03.308514    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f64def0-c36b-4a21-be70-77604220eaf1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-57s2t\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"
Oct 30 16:58:03.308868 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:03.308542    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q58h5\" (UniqueName: \"kubernetes.io/projected/1f64def0-c36b-4a21-be70-77604220eaf1-kube-api-access-q58h5\") pod \"maas-default-gateway-openshift-default-6db586dd6c-57s2t\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"
Oct 30 16:58:03.308868 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:03.308710    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f64def0-c36b-4a21-be70-77604220eaf1-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-57s2t\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"
Oct 30 16:58:03.308868 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:03.308757    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f64def0-c36b-4a21-be70-77604220eaf1-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-57s2t\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"
Oct 30 16:58:03.308868 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:03.308848    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f64def0-c36b-4a21-be70-77604220eaf1-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-57s2t\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"
Oct 30 16:58:03.308868 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:03.308860    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f64def0-c36b-4a21-be70-77604220eaf1-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-57s2t\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"
Oct 30 16:58:03.309243 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:03.309220    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f64def0-c36b-4a21-be70-77604220eaf1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-57s2t\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"
Oct 30 16:58:03.310870 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:03.310838    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f64def0-c36b-4a21-be70-77604220eaf1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-57s2t\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"
Oct 30 16:58:03.311049 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:03.311031    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f64def0-c36b-4a21-be70-77604220eaf1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-57s2t\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"
Oct 30 16:58:03.331250 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:03.331222    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f64def0-c36b-4a21-be70-77604220eaf1-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-57s2t\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"
Oct 30 16:58:03.331366 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:03.331348    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q58h5\" (UniqueName: \"kubernetes.io/projected/1f64def0-c36b-4a21-be70-77604220eaf1-kube-api-access-q58h5\") pod \"maas-default-gateway-openshift-default-6db586dd6c-57s2t\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"
Oct 30 16:58:03.446277 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:03.446236    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"
Oct 30 16:58:03.760811 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:03.760741    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"]
Oct 30 16:58:03.764035 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:58:03.764006    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f64def0_c36b_4a21_be70_77604220eaf1.slice/crio-874ef052aa766628f55851479e8542eea54c3110c2b56fe3506cc6d13965b47e WatchSource:0}: Error finding container 874ef052aa766628f55851479e8542eea54c3110c2b56fe3506cc6d13965b47e: Status 404 returned error can't find the container with id 874ef052aa766628f55851479e8542eea54c3110c2b56fe3506cc6d13965b47e
Oct 30 16:58:03.766099 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:03.766072    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 16:58:03.766155 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:03.766136    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 16:58:03.766189 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:03.766165    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 16:58:04.580720 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:04.580685    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t" event={"ID":"1f64def0-c36b-4a21-be70-77604220eaf1","Type":"ContainerStarted","Data":"7066935a5f5a1b936b007afbfc183f8f2376ccb73b7ea7772d07c7215ed853f7"}
Oct 30 16:58:04.581190 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:04.580739    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t" event={"ID":"1f64def0-c36b-4a21-be70-77604220eaf1","Type":"ContainerStarted","Data":"874ef052aa766628f55851479e8542eea54c3110c2b56fe3506cc6d13965b47e"}
Oct 30 16:58:04.636677 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:04.636627    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t" podStartSLOduration=1.636613993 podStartE2EDuration="1.636613993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:58:04.63599633 +0000 UTC m=+661.070898535" watchObservedRunningTime="2026-10-30 16:58:04.636613993 +0000 UTC m=+661.071516198"
Oct 30 16:58:05.447485 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:05.447450    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"
Oct 30 16:58:05.448949 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:05.448923    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-57s2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 16:58:05.449074 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:05.448969    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t" podUID="1f64def0-c36b-4a21-be70-77604220eaf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 16:58:06.447367 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:06.447339    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-57s2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 16:58:06.447862 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:06.447385    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t" podUID="1f64def0-c36b-4a21-be70-77604220eaf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 16:58:07.447203 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:07.447163    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-57s2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 16:58:07.447445 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:07.447225    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t" podUID="1f64def0-c36b-4a21-be70-77604220eaf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 16:58:08.447271 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:08.447233    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-57s2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 16:58:08.447479 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:08.447286    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t" podUID="1f64def0-c36b-4a21-be70-77604220eaf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 16:58:09.447350 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:09.447311    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-57s2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 16:58:09.447550 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:09.447381    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t" podUID="1f64def0-c36b-4a21-be70-77604220eaf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 16:58:10.447168 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:10.447133    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-57s2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 16:58:10.447347 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:10.447211    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t" podUID="1f64def0-c36b-4a21-be70-77604220eaf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 16:58:11.907782 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:11.907745    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"
Oct 30 16:58:11.908322 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:11.908303    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"
Oct 30 16:58:11.909243 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:58:11.909217    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"
Oct 30 16:58:26.977936 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:58:26.977895    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 16:58:56.978350 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:58:56.978312    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 16:59:03.553809 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:03.553773    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 16:59:08.616571 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:08.616527    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 16:59:09.787659 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:09.787624    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9"]
Oct 30 16:59:09.791418 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:09.791386    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9"
Oct 30 16:59:09.807982 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:09.807954    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 16:59:09.819704 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:09.819670    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-d384dc-serving-cert\""
Oct 30 16:59:09.819840 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:09.819709    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 16:59:09.819840 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:09.819740    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 16:59:09.819840 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:09.819783    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cdj9d\""
Oct 30 16:59:09.834552 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:09.834522    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/186d5a68-3c34-4953-8e2b-66a1278b9dd7-maas-api-tls\") pod \"maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9\" (UID: \"186d5a68-3c34-4953-8e2b-66a1278b9dd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9"
Oct 30 16:59:09.834663 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:09.834649    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/186d5a68-3c34-4953-8e2b-66a1278b9dd7-metrics-certs\") pod \"maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9\" (UID: \"186d5a68-3c34-4953-8e2b-66a1278b9dd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9"
Oct 30 16:59:09.834709 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:09.834682    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/186d5a68-3c34-4953-8e2b-66a1278b9dd7-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9\" (UID: \"186d5a68-3c34-4953-8e2b-66a1278b9dd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9"
Oct 30 16:59:09.834755 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:09.834721    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjvrm\" (UniqueName: \"kubernetes.io/projected/186d5a68-3c34-4953-8e2b-66a1278b9dd7-kube-api-access-wjvrm\") pod \"maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9\" (UID: \"186d5a68-3c34-4953-8e2b-66a1278b9dd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9"
Oct 30 16:59:09.846695 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:09.846665    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9"]
Oct 30 16:59:09.935430 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:09.935366    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjvrm\" (UniqueName: \"kubernetes.io/projected/186d5a68-3c34-4953-8e2b-66a1278b9dd7-kube-api-access-wjvrm\") pod \"maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9\" (UID: \"186d5a68-3c34-4953-8e2b-66a1278b9dd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9"
Oct 30 16:59:09.935642 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:09.935455    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/186d5a68-3c34-4953-8e2b-66a1278b9dd7-maas-api-tls\") pod \"maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9\" (UID: \"186d5a68-3c34-4953-8e2b-66a1278b9dd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9"
Oct 30 16:59:09.935642 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:09.935547    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/186d5a68-3c34-4953-8e2b-66a1278b9dd7-metrics-certs\") pod \"maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9\" (UID: \"186d5a68-3c34-4953-8e2b-66a1278b9dd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9"
Oct 30 16:59:09.935642 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:09.935583    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/186d5a68-3c34-4953-8e2b-66a1278b9dd7-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9\" (UID: \"186d5a68-3c34-4953-8e2b-66a1278b9dd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9"
Oct 30 16:59:09.936242 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:09.936215    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/186d5a68-3c34-4953-8e2b-66a1278b9dd7-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9\" (UID: \"186d5a68-3c34-4953-8e2b-66a1278b9dd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9"
Oct 30 16:59:09.937964 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:09.937940    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/186d5a68-3c34-4953-8e2b-66a1278b9dd7-maas-api-tls\") pod \"maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9\" (UID: \"186d5a68-3c34-4953-8e2b-66a1278b9dd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9"
Oct 30 16:59:09.938065 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:09.938002    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/186d5a68-3c34-4953-8e2b-66a1278b9dd7-metrics-certs\") pod \"maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9\" (UID: \"186d5a68-3c34-4953-8e2b-66a1278b9dd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9"
Oct 30 16:59:09.959024 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:09.958797    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjvrm\" (UniqueName: \"kubernetes.io/projected/186d5a68-3c34-4953-8e2b-66a1278b9dd7-kube-api-access-wjvrm\") pod \"maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9\" (UID: \"186d5a68-3c34-4953-8e2b-66a1278b9dd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9"
Oct 30 16:59:10.104857 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:10.104773    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9"
Oct 30 16:59:10.401798 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:10.401726    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9"]
Oct 30 16:59:10.405025 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:59:10.405002    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod186d5a68_3c34_4953_8e2b_66a1278b9dd7.slice/crio-17c9830c25f4150f83c599a0e116be6973c9d9febf898e0fe2152292060a01e0 WatchSource:0}: Error finding container 17c9830c25f4150f83c599a0e116be6973c9d9febf898e0fe2152292060a01e0: Status 404 returned error can't find the container with id 17c9830c25f4150f83c599a0e116be6973c9d9febf898e0fe2152292060a01e0
Oct 30 16:59:10.406308 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:10.406291    4433 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 16:59:10.871822 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:10.871777    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9" event={"ID":"186d5a68-3c34-4953-8e2b-66a1278b9dd7","Type":"ContainerStarted","Data":"17c9830c25f4150f83c599a0e116be6973c9d9febf898e0fe2152292060a01e0"}
Oct 30 16:59:12.883910 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:12.883872    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9" event={"ID":"186d5a68-3c34-4953-8e2b-66a1278b9dd7","Type":"ContainerStarted","Data":"c63cdc52fc4ae95864530e3e27cbd31a0c8fc67fc8079f78eb0c1c8db89e4fa9"}
Oct 30 16:59:12.884314 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:12.883941    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9"
Oct 30 16:59:12.929377 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:12.929313    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9" podStartSLOduration=1.622081794 podStartE2EDuration="3.929295073s" totalImagesPullingTime="2.307213279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:12.924954549 +0000 UTC m=+729.359856755" watchObservedRunningTime="2026-10-30 16:59:12.929295073 +0000 UTC m=+729.364197281"
Oct 30 16:59:14.260711 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:14.260671    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 16:59:18.894192 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:18.894160    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9"
Oct 30 16:59:19.474574 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:19.474543    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 16:59:21.815542 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:21.815503    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"]
Oct 30 16:59:21.819657 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:21.819635    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"
Oct 30 16:59:21.848440 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:21.845253    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-d384dc-openshift-default-dockercfg-r4qzb\""
Oct 30 16:59:21.855833 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:21.855723    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"]
Oct 30 16:59:21.958015 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:21.957973    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c690570f-1cba-450b-bcf6-0549e1ded4b7-istio-token\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"
Oct 30 16:59:21.958174 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:21.958033    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c690570f-1cba-450b-bcf6-0549e1ded4b7-istio-envoy\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"
Oct 30 16:59:21.958174 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:21.958056    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c690570f-1cba-450b-bcf6-0549e1ded4b7-istiod-ca-cert\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"
Oct 30 16:59:21.958174 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:21.958082    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c690570f-1cba-450b-bcf6-0549e1ded4b7-istio-podinfo\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"
Oct 30 16:59:21.958174 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:21.958107    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c690570f-1cba-450b-bcf6-0549e1ded4b7-workload-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"
Oct 30 16:59:21.958174 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:21.958142    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c690570f-1cba-450b-bcf6-0549e1ded4b7-credential-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"
Oct 30 16:59:21.958539 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:21.958180    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c690570f-1cba-450b-bcf6-0549e1ded4b7-workload-certs\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"
Oct 30 16:59:21.958539 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:21.958206    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7nxp\" (UniqueName: \"kubernetes.io/projected/c690570f-1cba-450b-bcf6-0549e1ded4b7-kube-api-access-v7nxp\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"
Oct 30 16:59:21.958539 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:21.958240    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c690570f-1cba-450b-bcf6-0549e1ded4b7-istio-data\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"
Oct 30 16:59:22.034522 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.034490    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"]
Oct 30 16:59:22.038093 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.038071    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"
Oct 30 16:59:22.060137 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.060087    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c690570f-1cba-450b-bcf6-0549e1ded4b7-istio-token\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"
Oct 30 16:59:22.060331 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.060159    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c690570f-1cba-450b-bcf6-0549e1ded4b7-istio-envoy\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"
Oct 30 16:59:22.060331 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.060190    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c690570f-1cba-450b-bcf6-0549e1ded4b7-istiod-ca-cert\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"
Oct 30 16:59:22.060331 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.060222    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c690570f-1cba-450b-bcf6-0549e1ded4b7-istio-podinfo\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"
Oct 30 16:59:22.060331 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.060249    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c690570f-1cba-450b-bcf6-0549e1ded4b7-workload-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"
Oct 30 16:59:22.060331 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.060290    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c690570f-1cba-450b-bcf6-0549e1ded4b7-credential-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"
Oct 30 16:59:22.060331 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.060332    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c690570f-1cba-450b-bcf6-0549e1ded4b7-workload-certs\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"
Oct 30 16:59:22.060647 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.060361    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7nxp\" (UniqueName: \"kubernetes.io/projected/c690570f-1cba-450b-bcf6-0549e1ded4b7-kube-api-access-v7nxp\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"
Oct 30 16:59:22.060647 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.060399    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c690570f-1cba-450b-bcf6-0549e1ded4b7-istio-data\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"
Oct 30 16:59:22.061057 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.061000    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c690570f-1cba-450b-bcf6-0549e1ded4b7-workload-certs\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"
Oct 30 16:59:22.061841 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.061815    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c690570f-1cba-450b-bcf6-0549e1ded4b7-istiod-ca-cert\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"
Oct 30 16:59:22.061970 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.061923    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c690570f-1cba-450b-bcf6-0549e1ded4b7-workload-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"
Oct 30 16:59:22.062247 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.062218    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c690570f-1cba-450b-bcf6-0549e1ded4b7-credential-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"
Oct 30 16:59:22.062915 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.062889    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c690570f-1cba-450b-bcf6-0549e1ded4b7-istio-data\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"
Oct 30 16:59:22.063019 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.062927    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"]
Oct 30 16:59:22.065005 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.064983    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c690570f-1cba-450b-bcf6-0549e1ded4b7-istio-envoy\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"
Oct 30 16:59:22.065160 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.065139    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c690570f-1cba-450b-bcf6-0549e1ded4b7-istio-podinfo\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"
Oct 30 16:59:22.084087 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.084021    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c690570f-1cba-450b-bcf6-0549e1ded4b7-istio-token\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"
Oct 30 16:59:22.084334 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.084309    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7nxp\" (UniqueName: \"kubernetes.io/projected/c690570f-1cba-450b-bcf6-0549e1ded4b7-kube-api-access-v7nxp\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"
Oct 30 16:59:22.084900 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.084880    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-b4f406-openshift-default-dockercfg-ftgfs\""
Oct 30 16:59:22.132386 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.132351    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"
Oct 30 16:59:22.163118 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.163077    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a71489d-ed10-4d2e-93df-d5770efaf301-istio-data\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"
Oct 30 16:59:22.163278 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.163130    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a71489d-ed10-4d2e-93df-d5770efaf301-istio-podinfo\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"
Oct 30 16:59:22.163330 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.163295    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a71489d-ed10-4d2e-93df-d5770efaf301-credential-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"
Oct 30 16:59:22.163366 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.163335    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a71489d-ed10-4d2e-93df-d5770efaf301-workload-certs\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"
Oct 30 16:59:22.163433 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.163364    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a71489d-ed10-4d2e-93df-d5770efaf301-istio-token\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"
Oct 30 16:59:22.163433 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.163393    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a71489d-ed10-4d2e-93df-d5770efaf301-istiod-ca-cert\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"
Oct 30 16:59:22.163560 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.163538    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a71489d-ed10-4d2e-93df-d5770efaf301-istio-envoy\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"
Oct 30 16:59:22.163601 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.163578    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a71489d-ed10-4d2e-93df-d5770efaf301-workload-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"
Oct 30 16:59:22.163666 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.163651    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq4b4\" (UniqueName: \"kubernetes.io/projected/1a71489d-ed10-4d2e-93df-d5770efaf301-kube-api-access-qq4b4\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"
Oct 30 16:59:22.264348 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.264312    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a71489d-ed10-4d2e-93df-d5770efaf301-credential-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"
Oct 30 16:59:22.264348 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.264352    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a71489d-ed10-4d2e-93df-d5770efaf301-workload-certs\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"
Oct 30 16:59:22.264601 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.264376    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a71489d-ed10-4d2e-93df-d5770efaf301-istio-token\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"
Oct 30 16:59:22.264601 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.264398    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a71489d-ed10-4d2e-93df-d5770efaf301-istiod-ca-cert\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"
Oct 30 16:59:22.264601 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.264467    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a71489d-ed10-4d2e-93df-d5770efaf301-istio-envoy\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"
Oct 30 16:59:22.264601 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.264493    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a71489d-ed10-4d2e-93df-d5770efaf301-workload-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"
Oct 30 16:59:22.264601 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.264561    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq4b4\" (UniqueName: \"kubernetes.io/projected/1a71489d-ed10-4d2e-93df-d5770efaf301-kube-api-access-qq4b4\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"
Oct 30 16:59:22.264848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.264617    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a71489d-ed10-4d2e-93df-d5770efaf301-istio-data\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"
Oct 30 16:59:22.264848 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.264642    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a71489d-ed10-4d2e-93df-d5770efaf301-istio-podinfo\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"
Oct 30 16:59:22.265084 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.265061    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a71489d-ed10-4d2e-93df-d5770efaf301-credential-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"
Oct 30 16:59:22.265363 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.265308    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a71489d-ed10-4d2e-93df-d5770efaf301-istiod-ca-cert\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"
Oct 30 16:59:22.265363 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.265339    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a71489d-ed10-4d2e-93df-d5770efaf301-workload-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"
Oct 30 16:59:22.265363 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.265318    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a71489d-ed10-4d2e-93df-d5770efaf301-istio-data\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"
Oct 30 16:59:22.265630 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.265562    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a71489d-ed10-4d2e-93df-d5770efaf301-workload-certs\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"
Oct 30 16:59:22.267130 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.267106    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a71489d-ed10-4d2e-93df-d5770efaf301-istio-podinfo\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"
Oct 30 16:59:22.267255 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.267150    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a71489d-ed10-4d2e-93df-d5770efaf301-istio-envoy\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"
Oct 30 16:59:22.288202 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.288171    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq4b4\" (UniqueName: \"kubernetes.io/projected/1a71489d-ed10-4d2e-93df-d5770efaf301-kube-api-access-qq4b4\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"
Oct 30 16:59:22.289652 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.289616    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a71489d-ed10-4d2e-93df-d5770efaf301-istio-token\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"
Oct 30 16:59:22.351661 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.351563    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"
Oct 30 16:59:22.453008 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.452951    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"]
Oct 30 16:59:22.456712 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:59:22.456680    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc690570f_1cba_450b_bcf6_0549e1ded4b7.slice/crio-0113dccd6661fc3cc4cf2bdd4dab369ae270075c5ee9a8680f764448efc62967 WatchSource:0}: Error finding container 0113dccd6661fc3cc4cf2bdd4dab369ae270075c5ee9a8680f764448efc62967: Status 404 returned error can't find the container with id 0113dccd6661fc3cc4cf2bdd4dab369ae270075c5ee9a8680f764448efc62967
Oct 30 16:59:22.459802 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.459767    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 16:59:22.459900 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.459860    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 16:59:22.459952 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.459903    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 16:59:22.608138 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.608048    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"]
Oct 30 16:59:22.614924 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.614895    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 16:59:22.630877 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.630848    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b80d83\"/\"openshift-service-ca.crt\""
Oct 30 16:59:22.631037 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.630945    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b80d83\"/\"kube-root-ca.crt\""
Oct 30 16:59:22.633505 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.633479    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"]
Oct 30 16:59:22.663474 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.663247    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"]
Oct 30 16:59:22.668982 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.668799    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6409a840-87ab-4b90-9955-a2a05c47b50f-tmp-dir\") pod \"simulator-w3-b80d83-kserve-694b784ccf-dwdq8\" (UID: \"6409a840-87ab-4b90-9955-a2a05c47b50f\") " pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 16:59:22.668982 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.668833    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6409a840-87ab-4b90-9955-a2a05c47b50f-tls-certs\") pod \"simulator-w3-b80d83-kserve-694b784ccf-dwdq8\" (UID: \"6409a840-87ab-4b90-9955-a2a05c47b50f\") " pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 16:59:22.668982 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.668904    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzhmc\" (UniqueName: \"kubernetes.io/projected/6409a840-87ab-4b90-9955-a2a05c47b50f-kube-api-access-fzhmc\") pod \"simulator-w3-b80d83-kserve-694b784ccf-dwdq8\" (UID: \"6409a840-87ab-4b90-9955-a2a05c47b50f\") " pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 16:59:22.668982 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.668937    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6409a840-87ab-4b90-9955-a2a05c47b50f-home\") pod \"simulator-w3-b80d83-kserve-694b784ccf-dwdq8\" (UID: \"6409a840-87ab-4b90-9955-a2a05c47b50f\") " pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 16:59:22.669274 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.669013    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6409a840-87ab-4b90-9955-a2a05c47b50f-model-cache\") pod \"simulator-w3-b80d83-kserve-694b784ccf-dwdq8\" (UID: \"6409a840-87ab-4b90-9955-a2a05c47b50f\") " pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 16:59:22.669274 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.669056    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6409a840-87ab-4b90-9955-a2a05c47b50f-dshm\") pod \"simulator-w3-b80d83-kserve-694b784ccf-dwdq8\" (UID: \"6409a840-87ab-4b90-9955-a2a05c47b50f\") " pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 16:59:22.670248 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:59:22.670222    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a71489d_ed10_4d2e_93df_d5770efaf301.slice/crio-a864a5bc3a1ae189bba195629fa3388081e291dfd5e6841c159ce93389717fd1 WatchSource:0}: Error finding container a864a5bc3a1ae189bba195629fa3388081e291dfd5e6841c159ce93389717fd1: Status 404 returned error can't find the container with id a864a5bc3a1ae189bba195629fa3388081e291dfd5e6841c159ce93389717fd1
Oct 30 16:59:22.676495 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.673432    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 16:59:22.676693 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.676663    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 16:59:22.676790 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.676767    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 16:59:22.728435 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.728216    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b80d83\"/\"simulator-w3-b80d83-kserve-self-signed-certs\""
Oct 30 16:59:22.728435 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.728397    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b80d83\"/\"default-dockercfg-pnr6k\""
Oct 30 16:59:22.771547 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.771511    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6409a840-87ab-4b90-9955-a2a05c47b50f-tmp-dir\") pod \"simulator-w3-b80d83-kserve-694b784ccf-dwdq8\" (UID: \"6409a840-87ab-4b90-9955-a2a05c47b50f\") " pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 16:59:22.771695 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.771560    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6409a840-87ab-4b90-9955-a2a05c47b50f-tls-certs\") pod \"simulator-w3-b80d83-kserve-694b784ccf-dwdq8\" (UID: \"6409a840-87ab-4b90-9955-a2a05c47b50f\") " pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 16:59:22.771695 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.771619    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzhmc\" (UniqueName: \"kubernetes.io/projected/6409a840-87ab-4b90-9955-a2a05c47b50f-kube-api-access-fzhmc\") pod \"simulator-w3-b80d83-kserve-694b784ccf-dwdq8\" (UID: \"6409a840-87ab-4b90-9955-a2a05c47b50f\") " pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 16:59:22.771695 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.771684    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6409a840-87ab-4b90-9955-a2a05c47b50f-home\") pod \"simulator-w3-b80d83-kserve-694b784ccf-dwdq8\" (UID: \"6409a840-87ab-4b90-9955-a2a05c47b50f\") " pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 16:59:22.772289 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.772265    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6409a840-87ab-4b90-9955-a2a05c47b50f-tmp-dir\") pod \"simulator-w3-b80d83-kserve-694b784ccf-dwdq8\" (UID: \"6409a840-87ab-4b90-9955-a2a05c47b50f\") " pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 16:59:22.772939 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.772918    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6409a840-87ab-4b90-9955-a2a05c47b50f-home\") pod \"simulator-w3-b80d83-kserve-694b784ccf-dwdq8\" (UID: \"6409a840-87ab-4b90-9955-a2a05c47b50f\") " pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 16:59:22.773724 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.773693    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6409a840-87ab-4b90-9955-a2a05c47b50f-model-cache\") pod \"simulator-w3-b80d83-kserve-694b784ccf-dwdq8\" (UID: \"6409a840-87ab-4b90-9955-a2a05c47b50f\") " pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 16:59:22.773821 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.773741    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6409a840-87ab-4b90-9955-a2a05c47b50f-dshm\") pod \"simulator-w3-b80d83-kserve-694b784ccf-dwdq8\" (UID: \"6409a840-87ab-4b90-9955-a2a05c47b50f\") " pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 16:59:22.774207 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.774189    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6409a840-87ab-4b90-9955-a2a05c47b50f-model-cache\") pod \"simulator-w3-b80d83-kserve-694b784ccf-dwdq8\" (UID: \"6409a840-87ab-4b90-9955-a2a05c47b50f\") " pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 16:59:22.779541 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.776737    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6409a840-87ab-4b90-9955-a2a05c47b50f-dshm\") pod \"simulator-w3-b80d83-kserve-694b784ccf-dwdq8\" (UID: \"6409a840-87ab-4b90-9955-a2a05c47b50f\") " pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 16:59:22.779541 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.777003    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6409a840-87ab-4b90-9955-a2a05c47b50f-tls-certs\") pod \"simulator-w3-b80d83-kserve-694b784ccf-dwdq8\" (UID: \"6409a840-87ab-4b90-9955-a2a05c47b50f\") " pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 16:59:22.796643 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.796611    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzhmc\" (UniqueName: \"kubernetes.io/projected/6409a840-87ab-4b90-9955-a2a05c47b50f-kube-api-access-fzhmc\") pod \"simulator-w3-b80d83-kserve-694b784ccf-dwdq8\" (UID: \"6409a840-87ab-4b90-9955-a2a05c47b50f\") " pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 16:59:22.931947 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.931384    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 16:59:22.935439 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.935368    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt" event={"ID":"1a71489d-ed10-4d2e-93df-d5770efaf301","Type":"ContainerStarted","Data":"4484f45df8f6f8358feb8d48a598cce5355e81ac1b1e0e7f1d07f0d1562f2210"}
Oct 30 16:59:22.935567 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.935464    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt" event={"ID":"1a71489d-ed10-4d2e-93df-d5770efaf301","Type":"ContainerStarted","Data":"a864a5bc3a1ae189bba195629fa3388081e291dfd5e6841c159ce93389717fd1"}
Oct 30 16:59:22.937485 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.937456    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6" event={"ID":"c690570f-1cba-450b-bcf6-0549e1ded4b7","Type":"ContainerStarted","Data":"f52b5c2fa4ff4776c5fa09f942903fc5f3104f65ab5fc44add042f79c1160ea6"}
Oct 30 16:59:22.937604 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.937509    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6" event={"ID":"c690570f-1cba-450b-bcf6-0549e1ded4b7","Type":"ContainerStarted","Data":"0113dccd6661fc3cc4cf2bdd4dab369ae270075c5ee9a8680f764448efc62967"}
Oct 30 16:59:22.995429 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:22.983033    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt" podStartSLOduration=0.983010862 podStartE2EDuration="983.010862ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:22.981652935 +0000 UTC m=+739.416555142" watchObservedRunningTime="2026-10-30 16:59:22.983010862 +0000 UTC m=+739.417913069"
Oct 30 16:59:23.031601 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:23.031540    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6" podStartSLOduration=2.031518375 podStartE2EDuration="2.031518375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:23.030236658 +0000 UTC m=+739.465138877" watchObservedRunningTime="2026-10-30 16:59:23.031518375 +0000 UTC m=+739.466420582"
Oct 30 16:59:23.133211 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:23.133173    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"
Oct 30 16:59:23.134774 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:23.134753    4433 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 16:59:23.134838 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:23.134798    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6" podUID="c690570f-1cba-450b-bcf6-0549e1ded4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 16:59:23.274914 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:23.274502    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"]
Oct 30 16:59:23.306967 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:59:23.306928    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6409a840_87ab_4b90_9955_a2a05c47b50f.slice/crio-4611bdec303155726d2741133d7b5b73b6535955782e7052622a3b87fc9d2bfc WatchSource:0}: Error finding container 4611bdec303155726d2741133d7b5b73b6535955782e7052622a3b87fc9d2bfc: Status 404 returned error can't find the container with id 4611bdec303155726d2741133d7b5b73b6535955782e7052622a3b87fc9d2bfc
Oct 30 16:59:23.371439 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:23.371285    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"
Oct 30 16:59:23.371663 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:23.371447    4433 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 16:59:23.371663 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:23.371490    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt" podUID="1a71489d-ed10-4d2e-93df-d5770efaf301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 16:59:23.947437 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:23.947307    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" event={"ID":"6409a840-87ab-4b90-9955-a2a05c47b50f","Type":"ContainerStarted","Data":"4611bdec303155726d2741133d7b5b73b6535955782e7052622a3b87fc9d2bfc"}
Oct 30 16:59:24.132816 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:24.132780    4433 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 16:59:24.133000 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:24.132839    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6" podUID="c690570f-1cba-450b-bcf6-0549e1ded4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 16:59:24.355477 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:24.354813    4433 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 16:59:24.355477 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:24.354872    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt" podUID="1a71489d-ed10-4d2e-93df-d5770efaf301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 16:59:25.133213 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:25.133177    4433 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 16:59:25.133754 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:25.133252    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6" podUID="c690570f-1cba-450b-bcf6-0549e1ded4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 16:59:25.370306 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:25.369838    4433 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 16:59:25.370306 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:25.369915    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt" podUID="1a71489d-ed10-4d2e-93df-d5770efaf301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 16:59:26.132697 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:26.132663    4433 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 16:59:26.132861 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:26.132715    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6" podUID="c690570f-1cba-450b-bcf6-0549e1ded4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 16:59:26.355457 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:26.354779    4433 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 16:59:26.355457 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:26.354841    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt" podUID="1a71489d-ed10-4d2e-93df-d5770efaf301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 16:59:27.011565 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:27.011505    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" event={"ID":"6409a840-87ab-4b90-9955-a2a05c47b50f","Type":"ContainerStarted","Data":"d123c293645970f64b703086d659a52d865fcdb1f3053d14bc8d3e36f5ea94ee"}
Oct 30 16:59:27.029001 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:59:27.028962    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 16:59:27.074429 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:27.069257    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podStartSLOduration=2.471764156 podStartE2EDuration="5.069235683s" totalImagesPullingTime="2.597471527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:27.067462088 +0000 UTC m=+743.502364306" watchObservedRunningTime="2026-10-30 16:59:27.069235683 +0000 UTC m=+743.504137893"
Oct 30 16:59:27.133270 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:27.133238    4433 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 16:59:27.133442 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:27.133290    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6" podUID="c690570f-1cba-450b-bcf6-0549e1ded4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 16:59:27.355004 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:27.354907    4433 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 16:59:27.355275 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:27.355232    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt" podUID="1a71489d-ed10-4d2e-93df-d5770efaf301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 16:59:27.932210 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:27.932174    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 16:59:28.132724 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:28.132691    4433 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 16:59:28.133808 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:28.133781    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6" podUID="c690570f-1cba-450b-bcf6-0549e1ded4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 16:59:28.353914 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:28.353801    4433 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 16:59:28.354194 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:28.354161    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt" podUID="1a71489d-ed10-4d2e-93df-d5770efaf301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 16:59:29.238529 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:29.238493    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"
Oct 30 16:59:29.238965 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:29.238912    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"
Oct 30 16:59:29.239633 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:29.239609    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"
Oct 30 16:59:29.357273 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:29.357217    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"
Oct 30 16:59:29.357771 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:29.357749    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"
Oct 30 16:59:29.358701 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:29.358677    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"
Oct 30 16:59:32.139882 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:32.139845    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 16:59:32.304063 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:32.304027    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"]
Oct 30 16:59:32.308693 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:32.308666    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 16:59:32.327437 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:32.327373    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"]
Oct 30 16:59:32.335902 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:32.335872    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-86f50a\"/\"openshift-service-ca.crt\""
Oct 30 16:59:32.335902 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:32.335873    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-86f50a\"/\"kube-root-ca.crt\""
Oct 30 16:59:32.366353 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:32.366316    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-86f50a\"/\"simulator-w5-86f50a-kserve-self-signed-certs\""
Oct 30 16:59:32.366353 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:32.366322    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-86f50a\"/\"default-dockercfg-tw9rv\""
Oct 30 16:59:32.440750 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:32.440709    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab427c7b-928a-4efd-a355-6633379fc574-home\") pod \"simulator-w5-86f50a-kserve-57c9654f4d-hp6pz\" (UID: \"ab427c7b-928a-4efd-a355-6633379fc574\") " pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 16:59:32.440959 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:32.440893    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab427c7b-928a-4efd-a355-6633379fc574-tmp-dir\") pod \"simulator-w5-86f50a-kserve-57c9654f4d-hp6pz\" (UID: \"ab427c7b-928a-4efd-a355-6633379fc574\") " pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 16:59:32.441063 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:32.441037    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab427c7b-928a-4efd-a355-6633379fc574-dshm\") pod \"simulator-w5-86f50a-kserve-57c9654f4d-hp6pz\" (UID: \"ab427c7b-928a-4efd-a355-6633379fc574\") " pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 16:59:32.441122 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:32.441091    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab427c7b-928a-4efd-a355-6633379fc574-tls-certs\") pod \"simulator-w5-86f50a-kserve-57c9654f4d-hp6pz\" (UID: \"ab427c7b-928a-4efd-a355-6633379fc574\") " pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 16:59:32.441219 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:32.441199    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4rt2\" (UniqueName: \"kubernetes.io/projected/ab427c7b-928a-4efd-a355-6633379fc574-kube-api-access-h4rt2\") pod \"simulator-w5-86f50a-kserve-57c9654f4d-hp6pz\" (UID: \"ab427c7b-928a-4efd-a355-6633379fc574\") " pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 16:59:32.441286 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:32.441265    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab427c7b-928a-4efd-a355-6633379fc574-model-cache\") pod \"simulator-w5-86f50a-kserve-57c9654f4d-hp6pz\" (UID: \"ab427c7b-928a-4efd-a355-6633379fc574\") " pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 16:59:32.542570 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:32.542534    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab427c7b-928a-4efd-a355-6633379fc574-tmp-dir\") pod \"simulator-w5-86f50a-kserve-57c9654f4d-hp6pz\" (UID: \"ab427c7b-928a-4efd-a355-6633379fc574\") " pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 16:59:32.542747 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:32.542609    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab427c7b-928a-4efd-a355-6633379fc574-dshm\") pod \"simulator-w5-86f50a-kserve-57c9654f4d-hp6pz\" (UID: \"ab427c7b-928a-4efd-a355-6633379fc574\") " pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 16:59:32.542747 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:32.542634    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab427c7b-928a-4efd-a355-6633379fc574-tls-certs\") pod \"simulator-w5-86f50a-kserve-57c9654f4d-hp6pz\" (UID: \"ab427c7b-928a-4efd-a355-6633379fc574\") " pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 16:59:32.542747 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:32.542668    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4rt2\" (UniqueName: \"kubernetes.io/projected/ab427c7b-928a-4efd-a355-6633379fc574-kube-api-access-h4rt2\") pod \"simulator-w5-86f50a-kserve-57c9654f4d-hp6pz\" (UID: \"ab427c7b-928a-4efd-a355-6633379fc574\") " pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 16:59:32.542747 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:32.542716    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab427c7b-928a-4efd-a355-6633379fc574-model-cache\") pod \"simulator-w5-86f50a-kserve-57c9654f4d-hp6pz\" (UID: \"ab427c7b-928a-4efd-a355-6633379fc574\") " pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 16:59:32.542747 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:32.542744    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab427c7b-928a-4efd-a355-6633379fc574-home\") pod \"simulator-w5-86f50a-kserve-57c9654f4d-hp6pz\" (UID: \"ab427c7b-928a-4efd-a355-6633379fc574\") " pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 16:59:32.543014 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:32.542986    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab427c7b-928a-4efd-a355-6633379fc574-tmp-dir\") pod \"simulator-w5-86f50a-kserve-57c9654f4d-hp6pz\" (UID: \"ab427c7b-928a-4efd-a355-6633379fc574\") " pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 16:59:32.543072 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:32.543025    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab427c7b-928a-4efd-a355-6633379fc574-model-cache\") pod \"simulator-w5-86f50a-kserve-57c9654f4d-hp6pz\" (UID: \"ab427c7b-928a-4efd-a355-6633379fc574\") " pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 16:59:32.543210 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:32.543104    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab427c7b-928a-4efd-a355-6633379fc574-home\") pod \"simulator-w5-86f50a-kserve-57c9654f4d-hp6pz\" (UID: \"ab427c7b-928a-4efd-a355-6633379fc574\") " pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 16:59:32.544847 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:32.544826    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab427c7b-928a-4efd-a355-6633379fc574-dshm\") pod \"simulator-w5-86f50a-kserve-57c9654f4d-hp6pz\" (UID: \"ab427c7b-928a-4efd-a355-6633379fc574\") " pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 16:59:32.545191 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:32.545167    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab427c7b-928a-4efd-a355-6633379fc574-tls-certs\") pod \"simulator-w5-86f50a-kserve-57c9654f4d-hp6pz\" (UID: \"ab427c7b-928a-4efd-a355-6633379fc574\") " pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 16:59:32.565657 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:32.565624    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4rt2\" (UniqueName: \"kubernetes.io/projected/ab427c7b-928a-4efd-a355-6633379fc574-kube-api-access-h4rt2\") pod \"simulator-w5-86f50a-kserve-57c9654f4d-hp6pz\" (UID: \"ab427c7b-928a-4efd-a355-6633379fc574\") " pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 16:59:32.621160 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:32.621117    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 16:59:32.927170 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:32.927122    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"]
Oct 30 16:59:32.931247 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:59:32.931219    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab427c7b_928a_4efd_a355_6633379fc574.slice/crio-a5c5a54fc77156e33710dca18399df33b926376b8f13c7ca2a2fde08a3fd6d2c WatchSource:0}: Error finding container a5c5a54fc77156e33710dca18399df33b926376b8f13c7ca2a2fde08a3fd6d2c: Status 404 returned error can't find the container with id a5c5a54fc77156e33710dca18399df33b926376b8f13c7ca2a2fde08a3fd6d2c
Oct 30 16:59:32.931797 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:32.931779    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 16:59:32.946145 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:32.946123    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 16:59:33.049530 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:33.049492    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" event={"ID":"ab427c7b-928a-4efd-a355-6633379fc574","Type":"ContainerStarted","Data":"dc996a98dac4fb8db5075b13e2ff966a2f239ac37a4ee1bd63e9d94cf3d76993"}
Oct 30 16:59:33.049720 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:33.049543    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" event={"ID":"ab427c7b-928a-4efd-a355-6633379fc574","Type":"ContainerStarted","Data":"a5c5a54fc77156e33710dca18399df33b926376b8f13c7ca2a2fde08a3fd6d2c"}
Oct 30 16:59:33.065280 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:33.065247    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 16:59:33.094782 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:33.094730    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podStartSLOduration=1.094716742 podStartE2EDuration="1.094716742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:33.094615985 +0000 UTC m=+749.529518209" watchObservedRunningTime="2026-10-30 16:59:33.094716742 +0000 UTC m=+749.529618948"
Oct 30 16:59:33.621566 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:33.621530    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 16:59:36.349885 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:36.349852    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"]
Oct 30 16:59:36.353917 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:36.353886    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 16:59:36.380163 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:36.380105    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-096d75\"/\"default-dockercfg-m777j\""
Oct 30 16:59:36.380163 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:36.380116    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-096d75\"/\"openshift-service-ca.crt\""
Oct 30 16:59:36.380163 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:36.380116    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-096d75\"/\"simulator-w0-096d75-kserve-self-signed-certs\""
Oct 30 16:59:36.380482 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:36.380221    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-096d75\"/\"kube-root-ca.crt\""
Oct 30 16:59:36.383281 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:36.383253    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"]
Oct 30 16:59:36.383845 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:36.383810    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c99f0236-6725-41a4-abd1-0bc0d9573910-tmp-dir\") pod \"simulator-w0-096d75-kserve-5cf76ddfdd-c599z\" (UID: \"c99f0236-6725-41a4-abd1-0bc0d9573910\") " pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 16:59:36.383944 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:36.383872    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c99f0236-6725-41a4-abd1-0bc0d9573910-dshm\") pod \"simulator-w0-096d75-kserve-5cf76ddfdd-c599z\" (UID: \"c99f0236-6725-41a4-abd1-0bc0d9573910\") " pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 16:59:36.383999 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:36.383967    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz9cf\" (UniqueName: \"kubernetes.io/projected/c99f0236-6725-41a4-abd1-0bc0d9573910-kube-api-access-pz9cf\") pod \"simulator-w0-096d75-kserve-5cf76ddfdd-c599z\" (UID: \"c99f0236-6725-41a4-abd1-0bc0d9573910\") " pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 16:59:36.384062 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:36.383999    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c99f0236-6725-41a4-abd1-0bc0d9573910-home\") pod \"simulator-w0-096d75-kserve-5cf76ddfdd-c599z\" (UID: \"c99f0236-6725-41a4-abd1-0bc0d9573910\") " pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 16:59:36.384062 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:36.384023    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c99f0236-6725-41a4-abd1-0bc0d9573910-tls-certs\") pod \"simulator-w0-096d75-kserve-5cf76ddfdd-c599z\" (UID: \"c99f0236-6725-41a4-abd1-0bc0d9573910\") " pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 16:59:36.384167 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:36.384144    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c99f0236-6725-41a4-abd1-0bc0d9573910-model-cache\") pod \"simulator-w0-096d75-kserve-5cf76ddfdd-c599z\" (UID: \"c99f0236-6725-41a4-abd1-0bc0d9573910\") " pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 16:59:36.485464 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:36.485396    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c99f0236-6725-41a4-abd1-0bc0d9573910-tmp-dir\") pod \"simulator-w0-096d75-kserve-5cf76ddfdd-c599z\" (UID: \"c99f0236-6725-41a4-abd1-0bc0d9573910\") " pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 16:59:36.485654 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:36.485488    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c99f0236-6725-41a4-abd1-0bc0d9573910-dshm\") pod \"simulator-w0-096d75-kserve-5cf76ddfdd-c599z\" (UID: \"c99f0236-6725-41a4-abd1-0bc0d9573910\") " pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 16:59:36.485654 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:36.485531    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz9cf\" (UniqueName: \"kubernetes.io/projected/c99f0236-6725-41a4-abd1-0bc0d9573910-kube-api-access-pz9cf\") pod \"simulator-w0-096d75-kserve-5cf76ddfdd-c599z\" (UID: \"c99f0236-6725-41a4-abd1-0bc0d9573910\") " pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 16:59:36.485654 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:36.485560    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c99f0236-6725-41a4-abd1-0bc0d9573910-home\") pod \"simulator-w0-096d75-kserve-5cf76ddfdd-c599z\" (UID: \"c99f0236-6725-41a4-abd1-0bc0d9573910\") " pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 16:59:36.485654 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:36.485584    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c99f0236-6725-41a4-abd1-0bc0d9573910-tls-certs\") pod \"simulator-w0-096d75-kserve-5cf76ddfdd-c599z\" (UID: \"c99f0236-6725-41a4-abd1-0bc0d9573910\") " pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 16:59:36.485654 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:36.485640    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c99f0236-6725-41a4-abd1-0bc0d9573910-model-cache\") pod \"simulator-w0-096d75-kserve-5cf76ddfdd-c599z\" (UID: \"c99f0236-6725-41a4-abd1-0bc0d9573910\") " pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 16:59:36.485943 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:36.485910    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c99f0236-6725-41a4-abd1-0bc0d9573910-tmp-dir\") pod \"simulator-w0-096d75-kserve-5cf76ddfdd-c599z\" (UID: \"c99f0236-6725-41a4-abd1-0bc0d9573910\") " pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 16:59:36.485999 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:36.485952    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c99f0236-6725-41a4-abd1-0bc0d9573910-home\") pod \"simulator-w0-096d75-kserve-5cf76ddfdd-c599z\" (UID: \"c99f0236-6725-41a4-abd1-0bc0d9573910\") " pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 16:59:36.486085 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:36.486064    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c99f0236-6725-41a4-abd1-0bc0d9573910-model-cache\") pod \"simulator-w0-096d75-kserve-5cf76ddfdd-c599z\" (UID: \"c99f0236-6725-41a4-abd1-0bc0d9573910\") " pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 16:59:36.487839 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:36.487807    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c99f0236-6725-41a4-abd1-0bc0d9573910-dshm\") pod \"simulator-w0-096d75-kserve-5cf76ddfdd-c599z\" (UID: \"c99f0236-6725-41a4-abd1-0bc0d9573910\") " pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 16:59:36.488275 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:36.488242    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c99f0236-6725-41a4-abd1-0bc0d9573910-tls-certs\") pod \"simulator-w0-096d75-kserve-5cf76ddfdd-c599z\" (UID: \"c99f0236-6725-41a4-abd1-0bc0d9573910\") " pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 16:59:36.508688 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:36.508659    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz9cf\" (UniqueName: \"kubernetes.io/projected/c99f0236-6725-41a4-abd1-0bc0d9573910-kube-api-access-pz9cf\") pod \"simulator-w0-096d75-kserve-5cf76ddfdd-c599z\" (UID: \"c99f0236-6725-41a4-abd1-0bc0d9573910\") " pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 16:59:36.671996 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:36.671916    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 16:59:36.970665 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:36.970634    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"]
Oct 30 16:59:36.973847 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:59:36.973820    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc99f0236_6725_41a4_abd1_0bc0d9573910.slice/crio-e8948d5897b3b31987e2fef9f1794d70cb4b066f4e766d623a85f6acee5791c4 WatchSource:0}: Error finding container e8948d5897b3b31987e2fef9f1794d70cb4b066f4e766d623a85f6acee5791c4: Status 404 returned error can't find the container with id e8948d5897b3b31987e2fef9f1794d70cb4b066f4e766d623a85f6acee5791c4
Oct 30 16:59:37.072276 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:37.072235    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" event={"ID":"c99f0236-6725-41a4-abd1-0bc0d9573910","Type":"ContainerStarted","Data":"906ad64932fa3b5bdec53a115d4bf5e51a83cb4e350d21ca105c30ef0260476c"}
Oct 30 16:59:37.072276 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:37.072280    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" event={"ID":"c99f0236-6725-41a4-abd1-0bc0d9573910","Type":"ContainerStarted","Data":"e8948d5897b3b31987e2fef9f1794d70cb4b066f4e766d623a85f6acee5791c4"}
Oct 30 16:59:37.116853 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:37.116799    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podStartSLOduration=1.11677958 podStartE2EDuration="1.11677958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:37.116168822 +0000 UTC m=+753.551071028" watchObservedRunningTime="2026-10-30 16:59:37.11677958 +0000 UTC m=+753.551681785"
Oct 30 16:59:37.672113 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:37.672066    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 16:59:42.554008 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:42.553973    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 16:59:42.622030 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:42.621991    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 16:59:42.634487 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:42.634451    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 16:59:43.112961 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:43.112930    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 16:59:43.779381 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:43.779347    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"]
Oct 30 16:59:43.783555 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:43.783529    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 16:59:43.797570 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:43.797547    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b4f406\"/\"openshift-service-ca.crt\""
Oct 30 16:59:43.809108 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:43.809074    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b4f406\"/\"kube-root-ca.crt\""
Oct 30 16:59:43.809108 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:43.809107    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b4f406\"/\"default-dockercfg-gf4d8\""
Oct 30 16:59:43.809277 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:43.809143    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b4f406\"/\"simulator-w2-b4f406-kserve-self-signed-certs\""
Oct 30 16:59:43.811086 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:43.811069    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"]
Oct 30 16:59:43.861249 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:43.861207    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a27c69-89cd-4802-a397-eff106d58bd1-model-cache\") pod \"simulator-w2-b4f406-kserve-7fd7658bff-vpq4v\" (UID: \"71a27c69-89cd-4802-a397-eff106d58bd1\") " pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 16:59:43.861460 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:43.861316    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a27c69-89cd-4802-a397-eff106d58bd1-tmp-dir\") pod \"simulator-w2-b4f406-kserve-7fd7658bff-vpq4v\" (UID: \"71a27c69-89cd-4802-a397-eff106d58bd1\") " pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 16:59:43.861460 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:43.861441    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a27c69-89cd-4802-a397-eff106d58bd1-home\") pod \"simulator-w2-b4f406-kserve-7fd7658bff-vpq4v\" (UID: \"71a27c69-89cd-4802-a397-eff106d58bd1\") " pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 16:59:43.861580 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:43.861484    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a27c69-89cd-4802-a397-eff106d58bd1-tls-certs\") pod \"simulator-w2-b4f406-kserve-7fd7658bff-vpq4v\" (UID: \"71a27c69-89cd-4802-a397-eff106d58bd1\") " pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 16:59:43.861648 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:43.861620    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p9c6\" (UniqueName: \"kubernetes.io/projected/71a27c69-89cd-4802-a397-eff106d58bd1-kube-api-access-9p9c6\") pod \"simulator-w2-b4f406-kserve-7fd7658bff-vpq4v\" (UID: \"71a27c69-89cd-4802-a397-eff106d58bd1\") " pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 16:59:43.861745 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:43.861662    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a27c69-89cd-4802-a397-eff106d58bd1-dshm\") pod \"simulator-w2-b4f406-kserve-7fd7658bff-vpq4v\" (UID: \"71a27c69-89cd-4802-a397-eff106d58bd1\") " pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 16:59:43.962768 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:43.962726    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a27c69-89cd-4802-a397-eff106d58bd1-model-cache\") pod \"simulator-w2-b4f406-kserve-7fd7658bff-vpq4v\" (UID: \"71a27c69-89cd-4802-a397-eff106d58bd1\") " pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 16:59:43.962965 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:43.962786    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a27c69-89cd-4802-a397-eff106d58bd1-tmp-dir\") pod \"simulator-w2-b4f406-kserve-7fd7658bff-vpq4v\" (UID: \"71a27c69-89cd-4802-a397-eff106d58bd1\") " pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 16:59:43.962965 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:43.962826    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a27c69-89cd-4802-a397-eff106d58bd1-home\") pod \"simulator-w2-b4f406-kserve-7fd7658bff-vpq4v\" (UID: \"71a27c69-89cd-4802-a397-eff106d58bd1\") " pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 16:59:43.962965 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:43.962848    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a27c69-89cd-4802-a397-eff106d58bd1-tls-certs\") pod \"simulator-w2-b4f406-kserve-7fd7658bff-vpq4v\" (UID: \"71a27c69-89cd-4802-a397-eff106d58bd1\") " pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 16:59:43.962965 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:43.962906    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p9c6\" (UniqueName: \"kubernetes.io/projected/71a27c69-89cd-4802-a397-eff106d58bd1-kube-api-access-9p9c6\") pod \"simulator-w2-b4f406-kserve-7fd7658bff-vpq4v\" (UID: \"71a27c69-89cd-4802-a397-eff106d58bd1\") " pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 16:59:43.962965 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:43.962933    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a27c69-89cd-4802-a397-eff106d58bd1-dshm\") pod \"simulator-w2-b4f406-kserve-7fd7658bff-vpq4v\" (UID: \"71a27c69-89cd-4802-a397-eff106d58bd1\") " pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 16:59:43.963238 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:43.963176    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a27c69-89cd-4802-a397-eff106d58bd1-model-cache\") pod \"simulator-w2-b4f406-kserve-7fd7658bff-vpq4v\" (UID: \"71a27c69-89cd-4802-a397-eff106d58bd1\") " pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 16:59:43.963293 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:43.963247    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a27c69-89cd-4802-a397-eff106d58bd1-tmp-dir\") pod \"simulator-w2-b4f406-kserve-7fd7658bff-vpq4v\" (UID: \"71a27c69-89cd-4802-a397-eff106d58bd1\") " pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 16:59:43.963599 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:43.963573    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a27c69-89cd-4802-a397-eff106d58bd1-home\") pod \"simulator-w2-b4f406-kserve-7fd7658bff-vpq4v\" (UID: \"71a27c69-89cd-4802-a397-eff106d58bd1\") " pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 16:59:43.965293 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:43.965263    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a27c69-89cd-4802-a397-eff106d58bd1-dshm\") pod \"simulator-w2-b4f406-kserve-7fd7658bff-vpq4v\" (UID: \"71a27c69-89cd-4802-a397-eff106d58bd1\") " pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 16:59:43.965628 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:43.965609    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a27c69-89cd-4802-a397-eff106d58bd1-tls-certs\") pod \"simulator-w2-b4f406-kserve-7fd7658bff-vpq4v\" (UID: \"71a27c69-89cd-4802-a397-eff106d58bd1\") " pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 16:59:43.986256 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:43.986228    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p9c6\" (UniqueName: \"kubernetes.io/projected/71a27c69-89cd-4802-a397-eff106d58bd1-kube-api-access-9p9c6\") pod \"simulator-w2-b4f406-kserve-7fd7658bff-vpq4v\" (UID: \"71a27c69-89cd-4802-a397-eff106d58bd1\") " pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 16:59:44.096444 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:44.096325    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 16:59:44.409941 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:44.409908    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"]
Oct 30 16:59:44.412360 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:59:44.412329    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71a27c69_89cd_4802_a397_eff106d58bd1.slice/crio-c3fb3fa58cdf54cf5fb2952cf20b5526a111fef89172892ebf9fffbde40d61b6 WatchSource:0}: Error finding container c3fb3fa58cdf54cf5fb2952cf20b5526a111fef89172892ebf9fffbde40d61b6: Status 404 returned error can't find the container with id c3fb3fa58cdf54cf5fb2952cf20b5526a111fef89172892ebf9fffbde40d61b6
Oct 30 16:59:45.116593 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:45.116553    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" event={"ID":"71a27c69-89cd-4802-a397-eff106d58bd1","Type":"ContainerStarted","Data":"ca5cf1b5f8831002e8775ce57802cebbd35efedc93b6f0e86c9a0936b07b9ae2"}
Oct 30 16:59:45.116593 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:45.116605    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" event={"ID":"71a27c69-89cd-4802-a397-eff106d58bd1","Type":"ContainerStarted","Data":"c3fb3fa58cdf54cf5fb2952cf20b5526a111fef89172892ebf9fffbde40d61b6"}
Oct 30 16:59:45.159679 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:45.159613    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podStartSLOduration=2.159591519 podStartE2EDuration="2.159591519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:45.15685072 +0000 UTC m=+761.591752928" watchObservedRunningTime="2026-10-30 16:59:45.159591519 +0000 UTC m=+761.594493726"
Oct 30 16:59:46.096883 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:46.096806    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 16:59:46.672483 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:46.672448    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 16:59:46.685156 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:46.685122    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 16:59:47.136826 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:47.136797    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 16:59:52.179327 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:52.179294    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"]
Oct 30 16:59:52.183606 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:52.183589    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"
Oct 30 16:59:52.213178 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:52.213146    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"]
Oct 30 16:59:52.247089 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:52.247051    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03c0ed14-97bb-4f58-9529-19ee6549b8c9-istio-envoy\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"
Oct 30 16:59:52.247324 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:52.247213    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03c0ed14-97bb-4f58-9529-19ee6549b8c9-istio-token\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"
Oct 30 16:59:52.247324 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:52.247274    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c0ed14-97bb-4f58-9529-19ee6549b8c9-workload-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"
Oct 30 16:59:52.247507 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:52.247339    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03c0ed14-97bb-4f58-9529-19ee6549b8c9-workload-certs\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"
Oct 30 16:59:52.247507 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:52.247391    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngfqw\" (UniqueName: \"kubernetes.io/projected/03c0ed14-97bb-4f58-9529-19ee6549b8c9-kube-api-access-ngfqw\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"
Oct 30 16:59:52.247602 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:52.247509    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03c0ed14-97bb-4f58-9529-19ee6549b8c9-istio-data\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"
Oct 30 16:59:52.247602 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:52.247549    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03c0ed14-97bb-4f58-9529-19ee6549b8c9-istiod-ca-cert\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"
Oct 30 16:59:52.247602 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:52.247579    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03c0ed14-97bb-4f58-9529-19ee6549b8c9-istio-podinfo\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"
Oct 30 16:59:52.247899 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:52.247669    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c0ed14-97bb-4f58-9529-19ee6549b8c9-credential-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"
Oct 30 16:59:52.348523 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:52.348484    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03c0ed14-97bb-4f58-9529-19ee6549b8c9-istio-token\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"
Oct 30 16:59:52.348732 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:52.348543    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c0ed14-97bb-4f58-9529-19ee6549b8c9-workload-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"
Oct 30 16:59:52.348732 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:52.348565    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03c0ed14-97bb-4f58-9529-19ee6549b8c9-workload-certs\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"
Oct 30 16:59:52.348732 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:52.348588    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngfqw\" (UniqueName: \"kubernetes.io/projected/03c0ed14-97bb-4f58-9529-19ee6549b8c9-kube-api-access-ngfqw\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"
Oct 30 16:59:52.348732 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:52.348613    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03c0ed14-97bb-4f58-9529-19ee6549b8c9-istio-data\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"
Oct 30 16:59:52.348948 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:52.348739    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03c0ed14-97bb-4f58-9529-19ee6549b8c9-istiod-ca-cert\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"
Oct 30 16:59:52.348948 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:52.348777    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03c0ed14-97bb-4f58-9529-19ee6549b8c9-istio-podinfo\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"
Oct 30 16:59:52.348948 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:52.348802    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c0ed14-97bb-4f58-9529-19ee6549b8c9-credential-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"
Oct 30 16:59:52.348948 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:52.348838    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03c0ed14-97bb-4f58-9529-19ee6549b8c9-istio-envoy\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"
Oct 30 16:59:52.349150 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:52.349014    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03c0ed14-97bb-4f58-9529-19ee6549b8c9-workload-certs\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"
Oct 30 16:59:52.349150 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:52.349089    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c0ed14-97bb-4f58-9529-19ee6549b8c9-workload-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"
Oct 30 16:59:52.349150 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:52.349108    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03c0ed14-97bb-4f58-9529-19ee6549b8c9-istio-data\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"
Oct 30 16:59:52.349254 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:52.349176    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c0ed14-97bb-4f58-9529-19ee6549b8c9-credential-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"
Oct 30 16:59:52.349558 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:52.349527    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03c0ed14-97bb-4f58-9529-19ee6549b8c9-istiod-ca-cert\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"
Oct 30 16:59:52.351171 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:52.351145    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03c0ed14-97bb-4f58-9529-19ee6549b8c9-istio-envoy\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"
Oct 30 16:59:52.351394 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:52.351373    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03c0ed14-97bb-4f58-9529-19ee6549b8c9-istio-podinfo\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"
Oct 30 16:59:52.372531 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:52.372502    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03c0ed14-97bb-4f58-9529-19ee6549b8c9-istio-token\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"
Oct 30 16:59:52.372727 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:52.372705    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngfqw\" (UniqueName: \"kubernetes.io/projected/03c0ed14-97bb-4f58-9529-19ee6549b8c9-kube-api-access-ngfqw\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"
Oct 30 16:59:52.497441 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:52.497316    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"
Oct 30 16:59:52.833294 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:52.833213    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"]
Oct 30 16:59:52.836473 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:59:52.836445    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c0ed14_97bb_4f58_9529_19ee6549b8c9.slice/crio-aeeb013791033ba88ac7431be13fbb9e5e5da87b7d788b15149e6a894e078e10 WatchSource:0}: Error finding container aeeb013791033ba88ac7431be13fbb9e5e5da87b7d788b15149e6a894e078e10: Status 404 returned error can't find the container with id aeeb013791033ba88ac7431be13fbb9e5e5da87b7d788b15149e6a894e078e10
Oct 30 16:59:52.838783 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:52.838752    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 16:59:52.838863 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:52.838820    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 16:59:52.838863 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:52.838851    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 16:59:53.165807 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:53.165716    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"]
Oct 30 16:59:53.172183 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:53.172147    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8" event={"ID":"03c0ed14-97bb-4f58-9529-19ee6549b8c9","Type":"ContainerStarted","Data":"a9e257a60f17cda250fa3ca42e55323ee7656692b8f1f33c9c1c7074132e7d7d"}
Oct 30 16:59:53.172322 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:53.172238    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8" event={"ID":"03c0ed14-97bb-4f58-9529-19ee6549b8c9","Type":"ContainerStarted","Data":"aeeb013791033ba88ac7431be13fbb9e5e5da87b7d788b15149e6a894e078e10"}
Oct 30 16:59:53.172382 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:53.172331    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 16:59:53.184642 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:53.184613    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"]
Oct 30 16:59:53.192949 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:53.192919    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-04248e\"/\"default-dockercfg-hpx7q\""
Oct 30 16:59:53.193120 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:53.193081    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-04248e\"/\"kube-root-ca.crt\""
Oct 30 16:59:53.193205 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:53.193179    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-04248e\"/\"openshift-service-ca.crt\""
Oct 30 16:59:53.197180 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:53.197154    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-04248e\"/\"simulator-w1-04248e-kserve-self-signed-certs\""
Oct 30 16:59:53.221374 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:53.221318    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8" podStartSLOduration=1.221300348 podStartE2EDuration="1.221300348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:53.219312457 +0000 UTC m=+769.654214687" watchObservedRunningTime="2026-10-30 16:59:53.221300348 +0000 UTC m=+769.656202554"
Oct 30 16:59:53.269044 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:53.268515    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf43bf3d-f126-40a1-a24f-10214a85ceb8-dshm\") pod \"simulator-w1-04248e-kserve-8dd89fdf5-fw9dm\" (UID: \"bf43bf3d-f126-40a1-a24f-10214a85ceb8\") " pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 16:59:53.269044 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:53.268577    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdk6d\" (UniqueName: \"kubernetes.io/projected/bf43bf3d-f126-40a1-a24f-10214a85ceb8-kube-api-access-mdk6d\") pod \"simulator-w1-04248e-kserve-8dd89fdf5-fw9dm\" (UID: \"bf43bf3d-f126-40a1-a24f-10214a85ceb8\") " pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 16:59:53.269044 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:53.268639    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf43bf3d-f126-40a1-a24f-10214a85ceb8-tmp-dir\") pod \"simulator-w1-04248e-kserve-8dd89fdf5-fw9dm\" (UID: \"bf43bf3d-f126-40a1-a24f-10214a85ceb8\") " pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 16:59:53.269044 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:53.268709    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf43bf3d-f126-40a1-a24f-10214a85ceb8-model-cache\") pod \"simulator-w1-04248e-kserve-8dd89fdf5-fw9dm\" (UID: \"bf43bf3d-f126-40a1-a24f-10214a85ceb8\") " pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 16:59:53.269044 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:53.268823    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf43bf3d-f126-40a1-a24f-10214a85ceb8-home\") pod \"simulator-w1-04248e-kserve-8dd89fdf5-fw9dm\" (UID: \"bf43bf3d-f126-40a1-a24f-10214a85ceb8\") " pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 16:59:53.269044 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:53.268895    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf43bf3d-f126-40a1-a24f-10214a85ceb8-tls-certs\") pod \"simulator-w1-04248e-kserve-8dd89fdf5-fw9dm\" (UID: \"bf43bf3d-f126-40a1-a24f-10214a85ceb8\") " pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 16:59:53.381203 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:53.373971    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdk6d\" (UniqueName: \"kubernetes.io/projected/bf43bf3d-f126-40a1-a24f-10214a85ceb8-kube-api-access-mdk6d\") pod \"simulator-w1-04248e-kserve-8dd89fdf5-fw9dm\" (UID: \"bf43bf3d-f126-40a1-a24f-10214a85ceb8\") " pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 16:59:53.381203 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:53.374075    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf43bf3d-f126-40a1-a24f-10214a85ceb8-tmp-dir\") pod \"simulator-w1-04248e-kserve-8dd89fdf5-fw9dm\" (UID: \"bf43bf3d-f126-40a1-a24f-10214a85ceb8\") " pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 16:59:53.381203 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:53.374134    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf43bf3d-f126-40a1-a24f-10214a85ceb8-model-cache\") pod \"simulator-w1-04248e-kserve-8dd89fdf5-fw9dm\" (UID: \"bf43bf3d-f126-40a1-a24f-10214a85ceb8\") " pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 16:59:53.381203 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:53.374233    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf43bf3d-f126-40a1-a24f-10214a85ceb8-home\") pod \"simulator-w1-04248e-kserve-8dd89fdf5-fw9dm\" (UID: \"bf43bf3d-f126-40a1-a24f-10214a85ceb8\") " pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 16:59:53.381203 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:53.374274    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf43bf3d-f126-40a1-a24f-10214a85ceb8-tls-certs\") pod \"simulator-w1-04248e-kserve-8dd89fdf5-fw9dm\" (UID: \"bf43bf3d-f126-40a1-a24f-10214a85ceb8\") " pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 16:59:53.381203 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:53.374320    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf43bf3d-f126-40a1-a24f-10214a85ceb8-dshm\") pod \"simulator-w1-04248e-kserve-8dd89fdf5-fw9dm\" (UID: \"bf43bf3d-f126-40a1-a24f-10214a85ceb8\") " pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 16:59:53.381203 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:53.375094    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf43bf3d-f126-40a1-a24f-10214a85ceb8-model-cache\") pod \"simulator-w1-04248e-kserve-8dd89fdf5-fw9dm\" (UID: \"bf43bf3d-f126-40a1-a24f-10214a85ceb8\") " pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 16:59:53.381203 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:53.375698    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf43bf3d-f126-40a1-a24f-10214a85ceb8-tmp-dir\") pod \"simulator-w1-04248e-kserve-8dd89fdf5-fw9dm\" (UID: \"bf43bf3d-f126-40a1-a24f-10214a85ceb8\") " pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 16:59:53.381203 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:53.375996    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf43bf3d-f126-40a1-a24f-10214a85ceb8-home\") pod \"simulator-w1-04248e-kserve-8dd89fdf5-fw9dm\" (UID: \"bf43bf3d-f126-40a1-a24f-10214a85ceb8\") " pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 16:59:53.381203 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:53.379936    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf43bf3d-f126-40a1-a24f-10214a85ceb8-dshm\") pod \"simulator-w1-04248e-kserve-8dd89fdf5-fw9dm\" (UID: \"bf43bf3d-f126-40a1-a24f-10214a85ceb8\") " pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 16:59:53.383994 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:53.383967    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf43bf3d-f126-40a1-a24f-10214a85ceb8-tls-certs\") pod \"simulator-w1-04248e-kserve-8dd89fdf5-fw9dm\" (UID: \"bf43bf3d-f126-40a1-a24f-10214a85ceb8\") " pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 16:59:53.398063 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:53.398032    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdk6d\" (UniqueName: \"kubernetes.io/projected/bf43bf3d-f126-40a1-a24f-10214a85ceb8-kube-api-access-mdk6d\") pod \"simulator-w1-04248e-kserve-8dd89fdf5-fw9dm\" (UID: \"bf43bf3d-f126-40a1-a24f-10214a85ceb8\") " pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 16:59:53.487344 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:53.487305    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 16:59:53.498669 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:53.498516    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"
Oct 30 16:59:53.500110 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:53.500088    4433 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 30 16:59:53.500189 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:53.500134    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8" podUID="03c0ed14-97bb-4f58-9529-19ee6549b8c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 30 16:59:53.788520 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:53.788397    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"]
Oct 30 16:59:53.793265 ip-10-0-131-13 kubenswrapper[4433]: W1030 16:59:53.793233    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf43bf3d_f126_40a1_a24f_10214a85ceb8.slice/crio-61b3dea60fa2b1105c8e9a1b2f0427412bb090f98cb59bcdb12dda80ca58f5a4 WatchSource:0}: Error finding container 61b3dea60fa2b1105c8e9a1b2f0427412bb090f98cb59bcdb12dda80ca58f5a4: Status 404 returned error can't find the container with id 61b3dea60fa2b1105c8e9a1b2f0427412bb090f98cb59bcdb12dda80ca58f5a4
Oct 30 16:59:54.097098 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:54.097010    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 16:59:54.109552 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:54.109523    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 16:59:54.182348 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:54.182302    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm" event={"ID":"bf43bf3d-f126-40a1-a24f-10214a85ceb8","Type":"ContainerStarted","Data":"0c6dc9d532bddec98d599936fc232a76278fd05ba4be5b2bf00477cab597be0a"}
Oct 30 16:59:54.182348 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:54.182353    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm" event={"ID":"bf43bf3d-f126-40a1-a24f-10214a85ceb8","Type":"ContainerStarted","Data":"61b3dea60fa2b1105c8e9a1b2f0427412bb090f98cb59bcdb12dda80ca58f5a4"}
Oct 30 16:59:54.193877 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:54.193850    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 16:59:54.228355 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:54.228307    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm" podStartSLOduration=1.228292697 podStartE2EDuration="1.228292697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:54.227923526 +0000 UTC m=+770.662825731" watchObservedRunningTime="2026-10-30 16:59:54.228292697 +0000 UTC m=+770.663194960"
Oct 30 16:59:54.488108 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:54.488070    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 16:59:54.498586 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:54.498558    4433 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 30 16:59:54.498668 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:54.498622    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8" podUID="03c0ed14-97bb-4f58-9529-19ee6549b8c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 30 16:59:55.498083 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:55.498053    4433 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 30 16:59:55.498502 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:55.498104    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8" podUID="03c0ed14-97bb-4f58-9529-19ee6549b8c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 30 16:59:56.497995 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:56.497958    4433 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 30 16:59:56.498152 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:56.498013    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8" podUID="03c0ed14-97bb-4f58-9529-19ee6549b8c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 30 16:59:57.003324 ip-10-0-131-13 kubenswrapper[4433]: E1030 16:59:57.003279    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 16:59:57.498373 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:57.498336    4433 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 30 16:59:57.498870 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:57.498389    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8" podUID="03c0ed14-97bb-4f58-9529-19ee6549b8c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 30 16:59:58.498686 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:58.498648    4433 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 30 16:59:58.499061 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:58.498704    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8" podUID="03c0ed14-97bb-4f58-9529-19ee6549b8c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 30 16:59:59.245578 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:59.245541    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 16:59:59.503288 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:59.503211    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"
Oct 30 16:59:59.503762 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:59.503741    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"
Oct 30 16:59:59.504632 ip-10-0-131-13 kubenswrapper[4433]: I1030 16:59:59.504611    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"
Oct 30 17:00:00.183700 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.183652    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889660-2rhzd"]
Oct 30 17:00:00.189330 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.189307    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-2rhzd"
Oct 30 17:00:00.192016 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.191992    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889660-5kzmn"]
Oct 30 17:00:00.195504 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.195482    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889660-5kzmn"
Oct 30 17:00:00.207313 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.207292    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\""
Oct 30 17:00:00.211219 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.211198    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889660-2rhzd"]
Oct 30 17:00:00.226107 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.226076    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889660-zst7z"]
Oct 30 17:00:00.230473 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.230448    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889660-5kzmn"]
Oct 30 17:00:00.230607 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.230482    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889660-sdwnr"]
Oct 30 17:00:00.230607 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.230561    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889660-zst7z"
Oct 30 17:00:00.244636 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.244609    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889660-zst7z"]
Oct 30 17:00:00.244796 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.244730    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889660-sdwnr"
Oct 30 17:00:00.250352 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.250279    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87888b19-88eb-4c0b-96fd-53bf3f0dedb6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889660-2rhzd\" (UID: \"87888b19-88eb-4c0b-96fd-53bf3f0dedb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-2rhzd"
Oct 30 17:00:00.250352 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.250343    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcbk8\" (UniqueName: \"kubernetes.io/projected/87888b19-88eb-4c0b-96fd-53bf3f0dedb6-kube-api-access-rcbk8\") pod \"maas-api-key-cleanup-29889660-2rhzd\" (UID: \"87888b19-88eb-4c0b-96fd-53bf3f0dedb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-2rhzd"
Oct 30 17:00:00.250702 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.250496    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6pb8\" (UniqueName: \"kubernetes.io/projected/019be6f3-aa64-4bee-955c-31bb2e24af98-kube-api-access-p6pb8\") pod \"maas-api-key-cleanup-e2e-worker-w0-096d75-29889660-5kzmn\" (UID: \"019be6f3-aa64-4bee-955c-31bb2e24af98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889660-5kzmn"
Oct 30 17:00:00.250702 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.250534    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/019be6f3-aa64-4bee-955c-31bb2e24af98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-096d75-29889660-5kzmn\" (UID: \"019be6f3-aa64-4bee-955c-31bb2e24af98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889660-5kzmn"
Oct 30 17:00:00.257933 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.257906    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889660-sdwnr"]
Oct 30 17:00:00.351363 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.351326    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-04248e-29889660-zst7z\" (UID: \"4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889660-zst7z"
Oct 30 17:00:00.351576 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.351417    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6pb8\" (UniqueName: \"kubernetes.io/projected/019be6f3-aa64-4bee-955c-31bb2e24af98-kube-api-access-p6pb8\") pod \"maas-api-key-cleanup-e2e-worker-w0-096d75-29889660-5kzmn\" (UID: \"019be6f3-aa64-4bee-955c-31bb2e24af98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889660-5kzmn"
Oct 30 17:00:00.351576 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.351484    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/019be6f3-aa64-4bee-955c-31bb2e24af98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-096d75-29889660-5kzmn\" (UID: \"019be6f3-aa64-4bee-955c-31bb2e24af98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889660-5kzmn"
Oct 30 17:00:00.351752 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.351732    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vdwv\" (UniqueName: \"kubernetes.io/projected/f41b1834-3bc9-467c-8fff-b7d2558b6271-kube-api-access-9vdwv\") pod \"maas-api-key-cleanup-e2e-worker-w5-86f50a-29889660-sdwnr\" (UID: \"f41b1834-3bc9-467c-8fff-b7d2558b6271\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889660-sdwnr"
Oct 30 17:00:00.351828 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.351773    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87888b19-88eb-4c0b-96fd-53bf3f0dedb6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889660-2rhzd\" (UID: \"87888b19-88eb-4c0b-96fd-53bf3f0dedb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-2rhzd"
Oct 30 17:00:00.351885 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.351829    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvxll\" (UniqueName: \"kubernetes.io/projected/4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220-kube-api-access-vvxll\") pod \"maas-api-key-cleanup-e2e-worker-w1-04248e-29889660-zst7z\" (UID: \"4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889660-zst7z"
Oct 30 17:00:00.351943 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.351880    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcbk8\" (UniqueName: \"kubernetes.io/projected/87888b19-88eb-4c0b-96fd-53bf3f0dedb6-kube-api-access-rcbk8\") pod \"maas-api-key-cleanup-29889660-2rhzd\" (UID: \"87888b19-88eb-4c0b-96fd-53bf3f0dedb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-2rhzd"
Oct 30 17:00:00.352066 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.352046    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f41b1834-3bc9-467c-8fff-b7d2558b6271-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-86f50a-29889660-sdwnr\" (UID: \"f41b1834-3bc9-467c-8fff-b7d2558b6271\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889660-sdwnr"
Oct 30 17:00:00.352271 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.352250    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/019be6f3-aa64-4bee-955c-31bb2e24af98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-096d75-29889660-5kzmn\" (UID: \"019be6f3-aa64-4bee-955c-31bb2e24af98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889660-5kzmn"
Oct 30 17:00:00.352515 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.352496    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87888b19-88eb-4c0b-96fd-53bf3f0dedb6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889660-2rhzd\" (UID: \"87888b19-88eb-4c0b-96fd-53bf3f0dedb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-2rhzd"
Oct 30 17:00:00.378165 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.378135    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcbk8\" (UniqueName: \"kubernetes.io/projected/87888b19-88eb-4c0b-96fd-53bf3f0dedb6-kube-api-access-rcbk8\") pod \"maas-api-key-cleanup-29889660-2rhzd\" (UID: \"87888b19-88eb-4c0b-96fd-53bf3f0dedb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-2rhzd"
Oct 30 17:00:00.381364 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.381340    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6pb8\" (UniqueName: \"kubernetes.io/projected/019be6f3-aa64-4bee-955c-31bb2e24af98-kube-api-access-p6pb8\") pod \"maas-api-key-cleanup-e2e-worker-w0-096d75-29889660-5kzmn\" (UID: \"019be6f3-aa64-4bee-955c-31bb2e24af98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889660-5kzmn"
Oct 30 17:00:00.452859 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.452774    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vdwv\" (UniqueName: \"kubernetes.io/projected/f41b1834-3bc9-467c-8fff-b7d2558b6271-kube-api-access-9vdwv\") pod \"maas-api-key-cleanup-e2e-worker-w5-86f50a-29889660-sdwnr\" (UID: \"f41b1834-3bc9-467c-8fff-b7d2558b6271\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889660-sdwnr"
Oct 30 17:00:00.452859 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.452831    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvxll\" (UniqueName: \"kubernetes.io/projected/4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220-kube-api-access-vvxll\") pod \"maas-api-key-cleanup-e2e-worker-w1-04248e-29889660-zst7z\" (UID: \"4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889660-zst7z"
Oct 30 17:00:00.453053 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.452906    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f41b1834-3bc9-467c-8fff-b7d2558b6271-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-86f50a-29889660-sdwnr\" (UID: \"f41b1834-3bc9-467c-8fff-b7d2558b6271\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889660-sdwnr"
Oct 30 17:00:00.453053 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.452944    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-04248e-29889660-zst7z\" (UID: \"4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889660-zst7z"
Oct 30 17:00:00.453644 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.453623    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-04248e-29889660-zst7z\" (UID: \"4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889660-zst7z"
Oct 30 17:00:00.453689 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.453623    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f41b1834-3bc9-467c-8fff-b7d2558b6271-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-86f50a-29889660-sdwnr\" (UID: \"f41b1834-3bc9-467c-8fff-b7d2558b6271\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889660-sdwnr"
Oct 30 17:00:00.476258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.476226    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvxll\" (UniqueName: \"kubernetes.io/projected/4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220-kube-api-access-vvxll\") pod \"maas-api-key-cleanup-e2e-worker-w1-04248e-29889660-zst7z\" (UID: \"4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889660-zst7z"
Oct 30 17:00:00.476425 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.476226    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vdwv\" (UniqueName: \"kubernetes.io/projected/f41b1834-3bc9-467c-8fff-b7d2558b6271-kube-api-access-9vdwv\") pod \"maas-api-key-cleanup-e2e-worker-w5-86f50a-29889660-sdwnr\" (UID: \"f41b1834-3bc9-467c-8fff-b7d2558b6271\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889660-sdwnr"
Oct 30 17:00:00.506168 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.506140    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-2rhzd"
Oct 30 17:00:00.512830 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.512808    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889660-5kzmn"
Oct 30 17:00:00.543843 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.543802    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889660-zst7z"
Oct 30 17:00:00.562263 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.562079    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889660-sdwnr"
Oct 30 17:00:00.856327 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.856151    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889660-5kzmn"]
Oct 30 17:00:00.858734 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:00:00.858704    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod019be6f3_aa64_4bee_955c_31bb2e24af98.slice/crio-4d30c883ae2084f84d9895c58e8f7767ea55b0eb234394567ddb4402fd185ad5 WatchSource:0}: Error finding container 4d30c883ae2084f84d9895c58e8f7767ea55b0eb234394567ddb4402fd185ad5: Status 404 returned error can't find the container with id 4d30c883ae2084f84d9895c58e8f7767ea55b0eb234394567ddb4402fd185ad5
Oct 30 17:00:00.863354 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.862192    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889660-2rhzd"]
Oct 30 17:00:00.864076 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:00:00.864047    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87888b19_88eb_4c0b_96fd_53bf3f0dedb6.slice/crio-d976d47bf0ed7b5ab4ef8c3bf3e97a18dd167301c33e8e8c72510a5006778480 WatchSource:0}: Error finding container d976d47bf0ed7b5ab4ef8c3bf3e97a18dd167301c33e8e8c72510a5006778480: Status 404 returned error can't find the container with id d976d47bf0ed7b5ab4ef8c3bf3e97a18dd167301c33e8e8c72510a5006778480
Oct 30 17:00:00.873796 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.873776    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889660-zst7z"]
Oct 30 17:00:00.876337 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:00:00.876311    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad253ce_bd3a_4f0b_ba6c_91ba08e5d220.slice/crio-4cd67f5cb321ea74587f6c1248668ae0948004d952ac185b9bb8a3c9bbc1ff09 WatchSource:0}: Error finding container 4cd67f5cb321ea74587f6c1248668ae0948004d952ac185b9bb8a3c9bbc1ff09: Status 404 returned error can't find the container with id 4cd67f5cb321ea74587f6c1248668ae0948004d952ac185b9bb8a3c9bbc1ff09
Oct 30 17:00:00.894091 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:00.894017    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889660-sdwnr"]
Oct 30 17:00:00.896812 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:00:00.896788    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf41b1834_3bc9_467c_8fff_b7d2558b6271.slice/crio-340396a7907a58fad8494cfe29945be3837f210c15d7bc49a4cb2e611a274e97 WatchSource:0}: Error finding container 340396a7907a58fad8494cfe29945be3837f210c15d7bc49a4cb2e611a274e97: Status 404 returned error can't find the container with id 340396a7907a58fad8494cfe29945be3837f210c15d7bc49a4cb2e611a274e97
Oct 30 17:00:01.221360 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:01.221327    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889660-sdwnr" event={"ID":"f41b1834-3bc9-467c-8fff-b7d2558b6271","Type":"ContainerStarted","Data":"340396a7907a58fad8494cfe29945be3837f210c15d7bc49a4cb2e611a274e97"}
Oct 30 17:00:01.222724 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:01.222693    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889660-5kzmn" event={"ID":"019be6f3-aa64-4bee-955c-31bb2e24af98","Type":"ContainerStarted","Data":"4d30c883ae2084f84d9895c58e8f7767ea55b0eb234394567ddb4402fd185ad5"}
Oct 30 17:00:01.223964 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:01.223937    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-2rhzd" event={"ID":"87888b19-88eb-4c0b-96fd-53bf3f0dedb6","Type":"ContainerStarted","Data":"d976d47bf0ed7b5ab4ef8c3bf3e97a18dd167301c33e8e8c72510a5006778480"}
Oct 30 17:00:01.225136 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:01.225113    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889660-zst7z" event={"ID":"4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220","Type":"ContainerStarted","Data":"4cd67f5cb321ea74587f6c1248668ae0948004d952ac185b9bb8a3c9bbc1ff09"}
Oct 30 17:00:03.488658 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:03.487861    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 17:00:03.506134 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:03.506105    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 17:00:04.265348 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:04.265302    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 17:00:06.264064 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:06.264025    4433 generic.go:356] "Generic (PLEG): container finished" podID="4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220" containerID="10bc2b4a4bffcf6228324512b814fe7b050d0a0800c11ddf07a48729dad7514a" exitCode=60
Oct 30 17:00:06.264572 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:06.264108    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889660-zst7z" event={"ID":"4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220","Type":"ContainerDied","Data":"10bc2b4a4bffcf6228324512b814fe7b050d0a0800c11ddf07a48729dad7514a"}
Oct 30 17:00:06.264572 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:06.264480    4433 scope.go:122] "RemoveContainer" containerID="10bc2b4a4bffcf6228324512b814fe7b050d0a0800c11ddf07a48729dad7514a"
Oct 30 17:00:06.265789 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:06.265768    4433 generic.go:356] "Generic (PLEG): container finished" podID="f41b1834-3bc9-467c-8fff-b7d2558b6271" containerID="8f5fdd962672852a3518d291db0a2220a639e6c3ff07f5ff15c6fced76c6d499" exitCode=60
Oct 30 17:00:06.265918 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:06.265791    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889660-sdwnr" event={"ID":"f41b1834-3bc9-467c-8fff-b7d2558b6271","Type":"ContainerDied","Data":"8f5fdd962672852a3518d291db0a2220a639e6c3ff07f5ff15c6fced76c6d499"}
Oct 30 17:00:06.266092 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:06.266073    4433 scope.go:122] "RemoveContainer" containerID="8f5fdd962672852a3518d291db0a2220a639e6c3ff07f5ff15c6fced76c6d499"
Oct 30 17:00:06.267442 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:06.267395    4433 generic.go:356] "Generic (PLEG): container finished" podID="019be6f3-aa64-4bee-955c-31bb2e24af98" containerID="1248c2c5a4bbc0d8adb7bd56760dbaa6f63ff31688d7638a847a15ee4c851ed7" exitCode=60
Oct 30 17:00:06.267545 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:06.267501    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889660-5kzmn" event={"ID":"019be6f3-aa64-4bee-955c-31bb2e24af98","Type":"ContainerDied","Data":"1248c2c5a4bbc0d8adb7bd56760dbaa6f63ff31688d7638a847a15ee4c851ed7"}
Oct 30 17:00:06.267779 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:06.267760    4433 scope.go:122] "RemoveContainer" containerID="1248c2c5a4bbc0d8adb7bd56760dbaa6f63ff31688d7638a847a15ee4c851ed7"
Oct 30 17:00:06.269252 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:06.269232    4433 generic.go:356] "Generic (PLEG): container finished" podID="87888b19-88eb-4c0b-96fd-53bf3f0dedb6" containerID="da8cb346bd4a584b3700988f291c3be670e08ca3ceb296296211adf6a9427bd8" exitCode=60
Oct 30 17:00:06.269326 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:06.269266    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-2rhzd" event={"ID":"87888b19-88eb-4c0b-96fd-53bf3f0dedb6","Type":"ContainerDied","Data":"da8cb346bd4a584b3700988f291c3be670e08ca3ceb296296211adf6a9427bd8"}
Oct 30 17:00:06.269518 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:06.269502    4433 scope.go:122] "RemoveContainer" containerID="da8cb346bd4a584b3700988f291c3be670e08ca3ceb296296211adf6a9427bd8"
Oct 30 17:00:07.275789 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:07.275757    4433 generic.go:356] "Generic (PLEG): container finished" podID="f41b1834-3bc9-467c-8fff-b7d2558b6271" containerID="2d3ab704a7e021fa7471c869b1b92a6347edf7ba44882636f03ebe524ea5b600" exitCode=60
Oct 30 17:00:07.276239 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:07.275826    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889660-sdwnr" event={"ID":"f41b1834-3bc9-467c-8fff-b7d2558b6271","Type":"ContainerDied","Data":"2d3ab704a7e021fa7471c869b1b92a6347edf7ba44882636f03ebe524ea5b600"}
Oct 30 17:00:07.276239 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:07.275900    4433 scope.go:122] "RemoveContainer" containerID="8f5fdd962672852a3518d291db0a2220a639e6c3ff07f5ff15c6fced76c6d499"
Oct 30 17:00:07.276239 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:07.276170    4433 scope.go:122] "RemoveContainer" containerID="2d3ab704a7e021fa7471c869b1b92a6347edf7ba44882636f03ebe524ea5b600"
Oct 30 17:00:07.276742 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:00:07.276523    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-86f50a-29889660-sdwnr_odh-ai-gateway-infra(f41b1834-3bc9-467c-8fff-b7d2558b6271)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889660-sdwnr" podUID="f41b1834-3bc9-467c-8fff-b7d2558b6271"
Oct 30 17:00:07.278029 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:07.278001    4433 generic.go:356] "Generic (PLEG): container finished" podID="019be6f3-aa64-4bee-955c-31bb2e24af98" containerID="fb60867e794cd9550c7f4c14adc7c539958e1febc7b57e5d5e04904185e26753" exitCode=60
Oct 30 17:00:07.278110 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:07.278077    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889660-5kzmn" event={"ID":"019be6f3-aa64-4bee-955c-31bb2e24af98","Type":"ContainerDied","Data":"fb60867e794cd9550c7f4c14adc7c539958e1febc7b57e5d5e04904185e26753"}
Oct 30 17:00:07.278337 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:07.278321    4433 scope.go:122] "RemoveContainer" containerID="fb60867e794cd9550c7f4c14adc7c539958e1febc7b57e5d5e04904185e26753"
Oct 30 17:00:07.278600 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:00:07.278579    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-096d75-29889660-5kzmn_odh-ai-gateway-infra(019be6f3-aa64-4bee-955c-31bb2e24af98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889660-5kzmn" podUID="019be6f3-aa64-4bee-955c-31bb2e24af98"
Oct 30 17:00:07.280103 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:07.280082    4433 generic.go:356] "Generic (PLEG): container finished" podID="87888b19-88eb-4c0b-96fd-53bf3f0dedb6" containerID="a8095eca7721e5c5d5a1aa3f5668f83d27da4488b3a6ef8b73707ef8396e19e1" exitCode=60
Oct 30 17:00:07.280185 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:07.280138    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-2rhzd" event={"ID":"87888b19-88eb-4c0b-96fd-53bf3f0dedb6","Type":"ContainerDied","Data":"a8095eca7721e5c5d5a1aa3f5668f83d27da4488b3a6ef8b73707ef8396e19e1"}
Oct 30 17:00:07.280430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:07.280416    4433 scope.go:122] "RemoveContainer" containerID="a8095eca7721e5c5d5a1aa3f5668f83d27da4488b3a6ef8b73707ef8396e19e1"
Oct 30 17:00:07.280664 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:00:07.280643    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889660-2rhzd_odh-ai-gateway-infra(87888b19-88eb-4c0b-96fd-53bf3f0dedb6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-2rhzd" podUID="87888b19-88eb-4c0b-96fd-53bf3f0dedb6"
Oct 30 17:00:07.282390 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:07.282373    4433 generic.go:356] "Generic (PLEG): container finished" podID="4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220" containerID="f7c765caf80a38e5cca6097d5ba342cfe7baef517fefa8693b7e30e362c0b940" exitCode=60
Oct 30 17:00:07.282518 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:07.282435    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889660-zst7z" event={"ID":"4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220","Type":"ContainerDied","Data":"f7c765caf80a38e5cca6097d5ba342cfe7baef517fefa8693b7e30e362c0b940"}
Oct 30 17:00:07.282744 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:07.282727    4433 scope.go:122] "RemoveContainer" containerID="f7c765caf80a38e5cca6097d5ba342cfe7baef517fefa8693b7e30e362c0b940"
Oct 30 17:00:07.283040 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:00:07.282978    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-04248e-29889660-zst7z_odh-ai-gateway-infra(4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889660-zst7z" podUID="4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220"
Oct 30 17:00:07.388338 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:07.388314    4433 scope.go:122] "RemoveContainer" containerID="1248c2c5a4bbc0d8adb7bd56760dbaa6f63ff31688d7638a847a15ee4c851ed7"
Oct 30 17:00:07.402023 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:07.401998    4433 scope.go:122] "RemoveContainer" containerID="da8cb346bd4a584b3700988f291c3be670e08ca3ceb296296211adf6a9427bd8"
Oct 30 17:00:07.414926 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:07.414904    4433 scope.go:122] "RemoveContainer" containerID="10bc2b4a4bffcf6228324512b814fe7b050d0a0800c11ddf07a48729dad7514a"
Oct 30 17:00:08.290219 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:08.290183    4433 scope.go:122] "RemoveContainer" containerID="a8095eca7721e5c5d5a1aa3f5668f83d27da4488b3a6ef8b73707ef8396e19e1"
Oct 30 17:00:08.290668 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:00:08.290537    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889660-2rhzd_odh-ai-gateway-infra(87888b19-88eb-4c0b-96fd-53bf3f0dedb6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-2rhzd" podUID="87888b19-88eb-4c0b-96fd-53bf3f0dedb6"
Oct 30 17:00:08.291903 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:08.291880    4433 scope.go:122] "RemoveContainer" containerID="f7c765caf80a38e5cca6097d5ba342cfe7baef517fefa8693b7e30e362c0b940"
Oct 30 17:00:08.292152 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:00:08.292128    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-04248e-29889660-zst7z_odh-ai-gateway-infra(4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889660-zst7z" podUID="4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220"
Oct 30 17:00:08.293540 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:08.293521    4433 scope.go:122] "RemoveContainer" containerID="2d3ab704a7e021fa7471c869b1b92a6347edf7ba44882636f03ebe524ea5b600"
Oct 30 17:00:08.293775 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:00:08.293753    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-86f50a-29889660-sdwnr_odh-ai-gateway-infra(f41b1834-3bc9-467c-8fff-b7d2558b6271)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889660-sdwnr" podUID="f41b1834-3bc9-467c-8fff-b7d2558b6271"
Oct 30 17:00:08.294977 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:08.294962    4433 scope.go:122] "RemoveContainer" containerID="fb60867e794cd9550c7f4c14adc7c539958e1febc7b57e5d5e04904185e26753"
Oct 30 17:00:08.295154 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:00:08.295138    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-096d75-29889660-5kzmn_odh-ai-gateway-infra(019be6f3-aa64-4bee-955c-31bb2e24af98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889660-5kzmn" podUID="019be6f3-aa64-4bee-955c-31bb2e24af98"
Oct 30 17:00:15.490685 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:15.490646    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:00:17.688859 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:17.688823    4433 scope.go:122] "RemoveContainer" containerID="2d3ab704a7e021fa7471c869b1b92a6347edf7ba44882636f03ebe524ea5b600"
Oct 30 17:00:18.348588 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:18.348496    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889660-sdwnr" event={"ID":"f41b1834-3bc9-467c-8fff-b7d2558b6271","Type":"ContainerStarted","Data":"e2813407774c749df90b51d39c9f42329c326b7f8f572504d38bd7c3aea9817d"}
Oct 30 17:00:18.390082 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:18.390025    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889660-sdwnr" podStartSLOduration=14.347847395 podStartE2EDuration="18.3900096s" totalImagesPullingTime="4.042162205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:18.388068668 +0000 UTC m=+794.822970873" watchObservedRunningTime="2026-10-30 17:00:18.3900096 +0000 UTC m=+794.824911806"
Oct 30 17:00:19.048936 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:19.048894    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"]
Oct 30 17:00:19.052867 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:19.052843    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:00:19.080116 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:19.080086    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-86f50a\"/\"premium-w5-86f50a-kserve-self-signed-certs\""
Oct 30 17:00:19.082951 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:19.082923    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"]
Oct 30 17:00:19.154823 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:19.154786    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e1669d0-fffa-471f-9275-de0fa70d98e7-model-cache\") pod \"premium-w5-86f50a-kserve-88687f89c-gpkw6\" (UID: \"2e1669d0-fffa-471f-9275-de0fa70d98e7\") " pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:00:19.154994 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:19.154856    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e1669d0-fffa-471f-9275-de0fa70d98e7-tls-certs\") pod \"premium-w5-86f50a-kserve-88687f89c-gpkw6\" (UID: \"2e1669d0-fffa-471f-9275-de0fa70d98e7\") " pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:00:19.154994 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:19.154962    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e1669d0-fffa-471f-9275-de0fa70d98e7-home\") pod \"premium-w5-86f50a-kserve-88687f89c-gpkw6\" (UID: \"2e1669d0-fffa-471f-9275-de0fa70d98e7\") " pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:00:19.155146 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:19.155117    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e1669d0-fffa-471f-9275-de0fa70d98e7-tmp-dir\") pod \"premium-w5-86f50a-kserve-88687f89c-gpkw6\" (UID: \"2e1669d0-fffa-471f-9275-de0fa70d98e7\") " pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:00:19.155211 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:19.155194    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e1669d0-fffa-471f-9275-de0fa70d98e7-dshm\") pod \"premium-w5-86f50a-kserve-88687f89c-gpkw6\" (UID: \"2e1669d0-fffa-471f-9275-de0fa70d98e7\") " pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:00:19.155249 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:19.155233    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl4ss\" (UniqueName: \"kubernetes.io/projected/2e1669d0-fffa-471f-9275-de0fa70d98e7-kube-api-access-vl4ss\") pod \"premium-w5-86f50a-kserve-88687f89c-gpkw6\" (UID: \"2e1669d0-fffa-471f-9275-de0fa70d98e7\") " pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:00:19.271214 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:19.256893    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e1669d0-fffa-471f-9275-de0fa70d98e7-tls-certs\") pod \"premium-w5-86f50a-kserve-88687f89c-gpkw6\" (UID: \"2e1669d0-fffa-471f-9275-de0fa70d98e7\") " pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:00:19.271214 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:19.256950    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e1669d0-fffa-471f-9275-de0fa70d98e7-home\") pod \"premium-w5-86f50a-kserve-88687f89c-gpkw6\" (UID: \"2e1669d0-fffa-471f-9275-de0fa70d98e7\") " pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:00:19.271214 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:19.257040    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e1669d0-fffa-471f-9275-de0fa70d98e7-tmp-dir\") pod \"premium-w5-86f50a-kserve-88687f89c-gpkw6\" (UID: \"2e1669d0-fffa-471f-9275-de0fa70d98e7\") " pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:00:19.271214 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:19.257097    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e1669d0-fffa-471f-9275-de0fa70d98e7-dshm\") pod \"premium-w5-86f50a-kserve-88687f89c-gpkw6\" (UID: \"2e1669d0-fffa-471f-9275-de0fa70d98e7\") " pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:00:19.271214 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:19.257129    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl4ss\" (UniqueName: \"kubernetes.io/projected/2e1669d0-fffa-471f-9275-de0fa70d98e7-kube-api-access-vl4ss\") pod \"premium-w5-86f50a-kserve-88687f89c-gpkw6\" (UID: \"2e1669d0-fffa-471f-9275-de0fa70d98e7\") " pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:00:19.271214 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:19.257197    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e1669d0-fffa-471f-9275-de0fa70d98e7-model-cache\") pod \"premium-w5-86f50a-kserve-88687f89c-gpkw6\" (UID: \"2e1669d0-fffa-471f-9275-de0fa70d98e7\") " pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:00:19.271214 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:19.257731    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e1669d0-fffa-471f-9275-de0fa70d98e7-model-cache\") pod \"premium-w5-86f50a-kserve-88687f89c-gpkw6\" (UID: \"2e1669d0-fffa-471f-9275-de0fa70d98e7\") " pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:00:19.271214 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:19.258116    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e1669d0-fffa-471f-9275-de0fa70d98e7-tmp-dir\") pod \"premium-w5-86f50a-kserve-88687f89c-gpkw6\" (UID: \"2e1669d0-fffa-471f-9275-de0fa70d98e7\") " pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:00:19.271214 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:19.261058    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e1669d0-fffa-471f-9275-de0fa70d98e7-tls-certs\") pod \"premium-w5-86f50a-kserve-88687f89c-gpkw6\" (UID: \"2e1669d0-fffa-471f-9275-de0fa70d98e7\") " pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:00:19.271214 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:19.261684    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e1669d0-fffa-471f-9275-de0fa70d98e7-home\") pod \"premium-w5-86f50a-kserve-88687f89c-gpkw6\" (UID: \"2e1669d0-fffa-471f-9275-de0fa70d98e7\") " pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:00:19.271214 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:19.267753    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e1669d0-fffa-471f-9275-de0fa70d98e7-dshm\") pod \"premium-w5-86f50a-kserve-88687f89c-gpkw6\" (UID: \"2e1669d0-fffa-471f-9275-de0fa70d98e7\") " pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:00:19.287050 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:19.285990    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl4ss\" (UniqueName: \"kubernetes.io/projected/2e1669d0-fffa-471f-9275-de0fa70d98e7-kube-api-access-vl4ss\") pod \"premium-w5-86f50a-kserve-88687f89c-gpkw6\" (UID: \"2e1669d0-fffa-471f-9275-de0fa70d98e7\") " pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:00:19.356696 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:19.356604    4433 generic.go:356] "Generic (PLEG): container finished" podID="f41b1834-3bc9-467c-8fff-b7d2558b6271" containerID="e2813407774c749df90b51d39c9f42329c326b7f8f572504d38bd7c3aea9817d" exitCode=60
Oct 30 17:00:19.356696 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:19.356679    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889660-sdwnr" event={"ID":"f41b1834-3bc9-467c-8fff-b7d2558b6271","Type":"ContainerDied","Data":"e2813407774c749df90b51d39c9f42329c326b7f8f572504d38bd7c3aea9817d"}
Oct 30 17:00:19.356907 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:19.356742    4433 scope.go:122] "RemoveContainer" containerID="2d3ab704a7e021fa7471c869b1b92a6347edf7ba44882636f03ebe524ea5b600"
Oct 30 17:00:19.357068 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:19.357050    4433 scope.go:122] "RemoveContainer" containerID="e2813407774c749df90b51d39c9f42329c326b7f8f572504d38bd7c3aea9817d"
Oct 30 17:00:19.357375 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:00:19.357350    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-86f50a-29889660-sdwnr_odh-ai-gateway-infra(f41b1834-3bc9-467c-8fff-b7d2558b6271)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889660-sdwnr" podUID="f41b1834-3bc9-467c-8fff-b7d2558b6271"
Oct 30 17:00:19.367907 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:19.367883    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:00:19.407042 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:19.407005    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889660-sdwnr"]
Oct 30 17:00:19.671352 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:19.671273    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"]
Oct 30 17:00:19.674996 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:00:19.674973    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e1669d0_fffa_471f_9275_de0fa70d98e7.slice/crio-d520cc4d3fc9703dda3911c0221c00aaec7c0c75e77f4cf0bec0aa3a179b5a64 WatchSource:0}: Error finding container d520cc4d3fc9703dda3911c0221c00aaec7c0c75e77f4cf0bec0aa3a179b5a64: Status 404 returned error can't find the container with id d520cc4d3fc9703dda3911c0221c00aaec7c0c75e77f4cf0bec0aa3a179b5a64
Oct 30 17:00:19.688303 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:19.688280    4433 scope.go:122] "RemoveContainer" containerID="a8095eca7721e5c5d5a1aa3f5668f83d27da4488b3a6ef8b73707ef8396e19e1"
Oct 30 17:00:20.363173 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:20.363078    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" event={"ID":"2e1669d0-fffa-471f-9275-de0fa70d98e7","Type":"ContainerStarted","Data":"f48a752853a7011ac6c56e390b455a70676c284ca85078d7cb6d29172b4af3c7"}
Oct 30 17:00:20.363173 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:20.363126    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" event={"ID":"2e1669d0-fffa-471f-9275-de0fa70d98e7","Type":"ContainerStarted","Data":"d520cc4d3fc9703dda3911c0221c00aaec7c0c75e77f4cf0bec0aa3a179b5a64"}
Oct 30 17:00:20.367121 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:20.367080    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-2rhzd" event={"ID":"87888b19-88eb-4c0b-96fd-53bf3f0dedb6","Type":"ContainerStarted","Data":"b1c5c0768feb214fa48031e943d18ddf9c1a23fc9feed4b9e70954598d86edfc"}
Oct 30 17:00:20.368464 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:20.368443    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:00:20.403972 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:20.403903    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" podStartSLOduration=1.403886088 podStartE2EDuration="1.403886088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:20.403268704 +0000 UTC m=+796.838170956" watchObservedRunningTime="2026-10-30 17:00:20.403886088 +0000 UTC m=+796.838788296"
Oct 30 17:00:20.443099 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:20.443051    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-2rhzd" podStartSLOduration=16.400308962 podStartE2EDuration="20.44303648s" totalImagesPullingTime="4.042727518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:20.442861117 +0000 UTC m=+796.877763322" watchObservedRunningTime="2026-10-30 17:00:20.44303648 +0000 UTC m=+796.877938685"
Oct 30 17:00:20.535310 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:20.535270    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889660-sdwnr"
Oct 30 17:00:20.671240 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:20.671153    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f41b1834-3bc9-467c-8fff-b7d2558b6271-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f41b1834-3bc9-467c-8fff-b7d2558b6271-maas-api-service-ca\") pod \"f41b1834-3bc9-467c-8fff-b7d2558b6271\" (UID: \"f41b1834-3bc9-467c-8fff-b7d2558b6271\") "
Oct 30 17:00:20.671240 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:20.671200    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f41b1834-3bc9-467c-8fff-b7d2558b6271-kube-api-access-9vdwv\" (UniqueName: \"kubernetes.io/projected/f41b1834-3bc9-467c-8fff-b7d2558b6271-kube-api-access-9vdwv\") pod \"f41b1834-3bc9-467c-8fff-b7d2558b6271\" (UID: \"f41b1834-3bc9-467c-8fff-b7d2558b6271\") "
Oct 30 17:00:20.671583 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:20.671560    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f41b1834-3bc9-467c-8fff-b7d2558b6271-maas-api-service-ca" pod "f41b1834-3bc9-467c-8fff-b7d2558b6271" (UID: "f41b1834-3bc9-467c-8fff-b7d2558b6271"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:00:20.673475 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:20.673426    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f41b1834-3bc9-467c-8fff-b7d2558b6271-kube-api-access-9vdwv" pod "f41b1834-3bc9-467c-8fff-b7d2558b6271" (UID: "f41b1834-3bc9-467c-8fff-b7d2558b6271"). InnerVolumeSpecName "kube-api-access-9vdwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:00:20.688457 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:20.688433    4433 scope.go:122] "RemoveContainer" containerID="f7c765caf80a38e5cca6097d5ba342cfe7baef517fefa8693b7e30e362c0b940"
Oct 30 17:00:20.772251 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:20.772213    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f41b1834-3bc9-467c-8fff-b7d2558b6271-maas-api-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:00:20.772251 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:20.772243    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vdwv\" (UniqueName: \"kubernetes.io/projected/f41b1834-3bc9-467c-8fff-b7d2558b6271-kube-api-access-9vdwv\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:00:21.373555 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:21.373471    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889660-sdwnr"
Oct 30 17:00:21.373555 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:21.373503    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889660-sdwnr" event={"ID":"f41b1834-3bc9-467c-8fff-b7d2558b6271","Type":"ContainerDied","Data":"340396a7907a58fad8494cfe29945be3837f210c15d7bc49a4cb2e611a274e97"}
Oct 30 17:00:21.373990 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:21.373565    4433 scope.go:122] "RemoveContainer" containerID="e2813407774c749df90b51d39c9f42329c326b7f8f572504d38bd7c3aea9817d"
Oct 30 17:00:21.376072 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:21.376044    4433 generic.go:356] "Generic (PLEG): container finished" podID="87888b19-88eb-4c0b-96fd-53bf3f0dedb6" containerID="b1c5c0768feb214fa48031e943d18ddf9c1a23fc9feed4b9e70954598d86edfc" exitCode=60
Oct 30 17:00:21.376251 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:21.376081    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-2rhzd" event={"ID":"87888b19-88eb-4c0b-96fd-53bf3f0dedb6","Type":"ContainerDied","Data":"b1c5c0768feb214fa48031e943d18ddf9c1a23fc9feed4b9e70954598d86edfc"}
Oct 30 17:00:21.376643 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:21.376625    4433 scope.go:122] "RemoveContainer" containerID="b1c5c0768feb214fa48031e943d18ddf9c1a23fc9feed4b9e70954598d86edfc"
Oct 30 17:00:21.376896 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:00:21.376877    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889660-2rhzd_odh-ai-gateway-infra(87888b19-88eb-4c0b-96fd-53bf3f0dedb6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-2rhzd" podUID="87888b19-88eb-4c0b-96fd-53bf3f0dedb6"
Oct 30 17:00:21.378083 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:21.378062    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889660-zst7z" event={"ID":"4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220","Type":"ContainerStarted","Data":"f17040b56afd8127803116436b4da93f12f72381fda135ee1cb1626f6e435461"}
Oct 30 17:00:21.383707 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:21.383688    4433 scope.go:122] "RemoveContainer" containerID="a8095eca7721e5c5d5a1aa3f5668f83d27da4488b3a6ef8b73707ef8396e19e1"
Oct 30 17:00:21.452010 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:21.451951    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889660-zst7z" podStartSLOduration=17.384699196 podStartE2EDuration="21.451935889s" totalImagesPullingTime="4.067236693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:21.427374628 +0000 UTC m=+797.862276836" watchObservedRunningTime="2026-10-30 17:00:21.451935889 +0000 UTC m=+797.886838094"
Oct 30 17:00:21.457124 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:21.453068    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889660-2rhzd"]
Oct 30 17:00:21.466500 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:21.466469    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889660-sdwnr"]
Oct 30 17:00:21.472686 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:21.472661    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889660-sdwnr"]
Oct 30 17:00:21.688675 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:21.688644    4433 scope.go:122] "RemoveContainer" containerID="fb60867e794cd9550c7f4c14adc7c539958e1febc7b57e5d5e04904185e26753"
Oct 30 17:00:21.695767 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:21.695741    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f41b1834-3bc9-467c-8fff-b7d2558b6271" path="/var/lib/kubelet/pods/f41b1834-3bc9-467c-8fff-b7d2558b6271/volumes"
Oct 30 17:00:22.388031 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:22.387940    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889660-5kzmn" event={"ID":"019be6f3-aa64-4bee-955c-31bb2e24af98","Type":"ContainerStarted","Data":"1d2ac1d29136578d13ab857c7dfcbf290009168272748fb6fcaa5b3f31552a67"}
Oct 30 17:00:22.391607 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:22.391576    4433 generic.go:356] "Generic (PLEG): container finished" podID="4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220" containerID="f17040b56afd8127803116436b4da93f12f72381fda135ee1cb1626f6e435461" exitCode=60
Oct 30 17:00:22.391702 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:22.391641    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889660-zst7z" event={"ID":"4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220","Type":"ContainerDied","Data":"f17040b56afd8127803116436b4da93f12f72381fda135ee1cb1626f6e435461"}
Oct 30 17:00:22.391702 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:22.391690    4433 scope.go:122] "RemoveContainer" containerID="f7c765caf80a38e5cca6097d5ba342cfe7baef517fefa8693b7e30e362c0b940"
Oct 30 17:00:22.392126 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:22.392109    4433 scope.go:122] "RemoveContainer" containerID="f17040b56afd8127803116436b4da93f12f72381fda135ee1cb1626f6e435461"
Oct 30 17:00:22.392378 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:00:22.392350    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-04248e-29889660-zst7z_odh-ai-gateway-infra(4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889660-zst7z" podUID="4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220"
Oct 30 17:00:22.433780 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:22.433703    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889660-5kzmn" podStartSLOduration=18.29728646 podStartE2EDuration="22.433682804s" totalImagesPullingTime="4.136396344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:22.431131078 +0000 UTC m=+798.866033284" watchObservedRunningTime="2026-10-30 17:00:22.433682804 +0000 UTC m=+798.868585018"
Oct 30 17:00:22.438696 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:22.438669    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889660-zst7z"]
Oct 30 17:00:22.538281 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:22.538256    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-2rhzd"
Oct 30 17:00:22.692823 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:22.692791    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87888b19-88eb-4c0b-96fd-53bf3f0dedb6-kube-api-access-rcbk8\" (UniqueName: \"kubernetes.io/projected/87888b19-88eb-4c0b-96fd-53bf3f0dedb6-kube-api-access-rcbk8\") pod \"87888b19-88eb-4c0b-96fd-53bf3f0dedb6\" (UID: \"87888b19-88eb-4c0b-96fd-53bf3f0dedb6\") "
Oct 30 17:00:22.693045 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:22.692881    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87888b19-88eb-4c0b-96fd-53bf3f0dedb6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87888b19-88eb-4c0b-96fd-53bf3f0dedb6-maas-api-service-ca\") pod \"87888b19-88eb-4c0b-96fd-53bf3f0dedb6\" (UID: \"87888b19-88eb-4c0b-96fd-53bf3f0dedb6\") "
Oct 30 17:00:22.693271 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:22.693247    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87888b19-88eb-4c0b-96fd-53bf3f0dedb6-maas-api-service-ca" pod "87888b19-88eb-4c0b-96fd-53bf3f0dedb6" (UID: "87888b19-88eb-4c0b-96fd-53bf3f0dedb6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:00:22.695040 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:22.695004    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87888b19-88eb-4c0b-96fd-53bf3f0dedb6-kube-api-access-rcbk8" pod "87888b19-88eb-4c0b-96fd-53bf3f0dedb6" (UID: "87888b19-88eb-4c0b-96fd-53bf3f0dedb6"). InnerVolumeSpecName "kube-api-access-rcbk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:00:22.794588 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:22.794544    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87888b19-88eb-4c0b-96fd-53bf3f0dedb6-maas-api-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:00:22.794588 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:22.794577    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcbk8\" (UniqueName: \"kubernetes.io/projected/87888b19-88eb-4c0b-96fd-53bf3f0dedb6-kube-api-access-rcbk8\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:00:23.401367 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:23.401341    4433 generic.go:356] "Generic (PLEG): container finished" podID="019be6f3-aa64-4bee-955c-31bb2e24af98" containerID="1d2ac1d29136578d13ab857c7dfcbf290009168272748fb6fcaa5b3f31552a67" exitCode=60
Oct 30 17:00:23.401776 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:23.401398    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889660-5kzmn" event={"ID":"019be6f3-aa64-4bee-955c-31bb2e24af98","Type":"ContainerDied","Data":"1d2ac1d29136578d13ab857c7dfcbf290009168272748fb6fcaa5b3f31552a67"}
Oct 30 17:00:23.401776 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:23.401473    4433 scope.go:122] "RemoveContainer" containerID="fb60867e794cd9550c7f4c14adc7c539958e1febc7b57e5d5e04904185e26753"
Oct 30 17:00:23.401776 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:23.401754    4433 scope.go:122] "RemoveContainer" containerID="1d2ac1d29136578d13ab857c7dfcbf290009168272748fb6fcaa5b3f31552a67"
Oct 30 17:00:23.402021 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:00:23.401995    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-096d75-29889660-5kzmn_odh-ai-gateway-infra(019be6f3-aa64-4bee-955c-31bb2e24af98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889660-5kzmn" podUID="019be6f3-aa64-4bee-955c-31bb2e24af98"
Oct 30 17:00:23.403035 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:23.403011    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-2rhzd" event={"ID":"87888b19-88eb-4c0b-96fd-53bf3f0dedb6","Type":"ContainerDied","Data":"d976d47bf0ed7b5ab4ef8c3bf3e97a18dd167301c33e8e8c72510a5006778480"}
Oct 30 17:00:23.403123 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:23.403055    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-2rhzd"
Oct 30 17:00:23.413799 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:23.413748    4433 scope.go:122] "RemoveContainer" containerID="b1c5c0768feb214fa48031e943d18ddf9c1a23fc9feed4b9e70954598d86edfc"
Oct 30 17:00:23.443757 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:23.443720    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889660-5kzmn"]
Oct 30 17:00:23.538759 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:23.538532    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889660-2rhzd"]
Oct 30 17:00:23.556625 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:23.547328    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889660-2rhzd"]
Oct 30 17:00:23.579761 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:23.579730    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889660-zst7z"
Oct 30 17:00:23.601927 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:23.601894    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220-kube-api-access-vvxll\" (UniqueName: \"kubernetes.io/projected/4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220-kube-api-access-vvxll\") pod \"4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220\" (UID: \"4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220\") "
Oct 30 17:00:23.602097 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:23.602072    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220-maas-api-service-ca\") pod \"4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220\" (UID: \"4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220\") "
Oct 30 17:00:23.602497 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:23.602468    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220-maas-api-service-ca" pod "4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220" (UID: "4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:00:23.603963 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:23.603939    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220-kube-api-access-vvxll" pod "4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220" (UID: "4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220"). InnerVolumeSpecName "kube-api-access-vvxll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:00:23.696370 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:23.696335    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87888b19-88eb-4c0b-96fd-53bf3f0dedb6" path="/var/lib/kubelet/pods/87888b19-88eb-4c0b-96fd-53bf3f0dedb6/volumes"
Oct 30 17:00:23.702854 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:23.702819    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220-maas-api-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:00:23.702854 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:23.702845    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvxll\" (UniqueName: \"kubernetes.io/projected/4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220-kube-api-access-vvxll\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:00:24.420940 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:24.420906    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889660-zst7z" event={"ID":"4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220","Type":"ContainerDied","Data":"4cd67f5cb321ea74587f6c1248668ae0948004d952ac185b9bb8a3c9bbc1ff09"}
Oct 30 17:00:24.421267 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:24.420953    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889660-zst7z"
Oct 30 17:00:24.421267 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:24.420999    4433 scope.go:122] "RemoveContainer" containerID="f17040b56afd8127803116436b4da93f12f72381fda135ee1cb1626f6e435461"
Oct 30 17:00:24.498537 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:24.496077    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889660-zst7z"]
Oct 30 17:00:24.499182 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:24.499150    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889660-zst7z"]
Oct 30 17:00:24.558792 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:24.558762    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889660-5kzmn"
Oct 30 17:00:24.619268 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:24.619229    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/019be6f3-aa64-4bee-955c-31bb2e24af98-kube-api-access-p6pb8\" (UniqueName: \"kubernetes.io/projected/019be6f3-aa64-4bee-955c-31bb2e24af98-kube-api-access-p6pb8\") pod \"019be6f3-aa64-4bee-955c-31bb2e24af98\" (UID: \"019be6f3-aa64-4bee-955c-31bb2e24af98\") "
Oct 30 17:00:24.619485 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:24.619357    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/019be6f3-aa64-4bee-955c-31bb2e24af98-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/019be6f3-aa64-4bee-955c-31bb2e24af98-maas-api-service-ca\") pod \"019be6f3-aa64-4bee-955c-31bb2e24af98\" (UID: \"019be6f3-aa64-4bee-955c-31bb2e24af98\") "
Oct 30 17:00:24.620089 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:24.620058    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/019be6f3-aa64-4bee-955c-31bb2e24af98-maas-api-service-ca" pod "019be6f3-aa64-4bee-955c-31bb2e24af98" (UID: "019be6f3-aa64-4bee-955c-31bb2e24af98"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:00:24.622157 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:24.622123    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/019be6f3-aa64-4bee-955c-31bb2e24af98-kube-api-access-p6pb8" pod "019be6f3-aa64-4bee-955c-31bb2e24af98" (UID: "019be6f3-aa64-4bee-955c-31bb2e24af98"). InnerVolumeSpecName "kube-api-access-p6pb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:00:24.721047 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:24.721009    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6pb8\" (UniqueName: \"kubernetes.io/projected/019be6f3-aa64-4bee-955c-31bb2e24af98-kube-api-access-p6pb8\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:00:24.721047 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:24.721042    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/019be6f3-aa64-4bee-955c-31bb2e24af98-maas-api-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:00:25.426503 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:25.426467    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889660-5kzmn"
Oct 30 17:00:25.426971 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:25.426469    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889660-5kzmn" event={"ID":"019be6f3-aa64-4bee-955c-31bb2e24af98","Type":"ContainerDied","Data":"4d30c883ae2084f84d9895c58e8f7767ea55b0eb234394567ddb4402fd185ad5"}
Oct 30 17:00:25.426971 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:25.426595    4433 scope.go:122] "RemoveContainer" containerID="1d2ac1d29136578d13ab857c7dfcbf290009168272748fb6fcaa5b3f31552a67"
Oct 30 17:00:25.475924 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:25.475893    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889660-5kzmn"]
Oct 30 17:00:25.487105 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:25.487075    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889660-5kzmn"]
Oct 30 17:00:25.695717 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:25.695639    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="019be6f3-aa64-4bee-955c-31bb2e24af98" path="/var/lib/kubelet/pods/019be6f3-aa64-4bee-955c-31bb2e24af98/volumes"
Oct 30 17:00:25.696087 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:25.696073    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220" path="/var/lib/kubelet/pods/4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220/volumes"
Oct 30 17:00:26.977395 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:00:26.977355    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:00:29.368106 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:29.368068    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:00:29.382145 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:29.382117    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:00:29.461079 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:29.461050    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:00:33.724858 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:33.724819    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:00:37.411053 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.411011    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"]
Oct 30 17:00:37.411577 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.411560    4433 cpu_manager.go:414] "containerMap: removing container" podUID="f41b1834-3bc9-467c-8fff-b7d2558b6271" containerName="cleanup"
Oct 30 17:00:37.411631 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.411579    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f41b1834-3bc9-467c-8fff-b7d2558b6271" containerName="cleanup"
Oct 30 17:00:37.411631 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.411589    4433 cpu_manager.go:414] "containerMap: removing container" podUID="87888b19-88eb-4c0b-96fd-53bf3f0dedb6" containerName="cleanup"
Oct 30 17:00:37.411631 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.411594    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87888b19-88eb-4c0b-96fd-53bf3f0dedb6" containerName="cleanup"
Oct 30 17:00:37.411631 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.411603    4433 cpu_manager.go:414] "containerMap: removing container" podUID="4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220" containerName="cleanup"
Oct 30 17:00:37.411631 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.411608    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220" containerName="cleanup"
Oct 30 17:00:37.411631 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.411614    4433 cpu_manager.go:414] "containerMap: removing container" podUID="f41b1834-3bc9-467c-8fff-b7d2558b6271" containerName="cleanup"
Oct 30 17:00:37.411631 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.411619    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f41b1834-3bc9-467c-8fff-b7d2558b6271" containerName="cleanup"
Oct 30 17:00:37.411631 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.411628    4433 cpu_manager.go:414] "containerMap: removing container" podUID="87888b19-88eb-4c0b-96fd-53bf3f0dedb6" containerName="cleanup"
Oct 30 17:00:37.411631 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.411632    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87888b19-88eb-4c0b-96fd-53bf3f0dedb6" containerName="cleanup"
Oct 30 17:00:37.411909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.411639    4433 cpu_manager.go:414] "containerMap: removing container" podUID="87888b19-88eb-4c0b-96fd-53bf3f0dedb6" containerName="cleanup"
Oct 30 17:00:37.411909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.411643    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87888b19-88eb-4c0b-96fd-53bf3f0dedb6" containerName="cleanup"
Oct 30 17:00:37.411909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.411652    4433 cpu_manager.go:414] "containerMap: removing container" podUID="4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220" containerName="cleanup"
Oct 30 17:00:37.411909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.411656    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220" containerName="cleanup"
Oct 30 17:00:37.411909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.411668    4433 cpu_manager.go:414] "containerMap: removing container" podUID="019be6f3-aa64-4bee-955c-31bb2e24af98" containerName="cleanup"
Oct 30 17:00:37.411909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.411673    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="019be6f3-aa64-4bee-955c-31bb2e24af98" containerName="cleanup"
Oct 30 17:00:37.411909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.411679    4433 cpu_manager.go:414] "containerMap: removing container" podUID="4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220" containerName="cleanup"
Oct 30 17:00:37.411909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.411683    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220" containerName="cleanup"
Oct 30 17:00:37.411909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.411691    4433 cpu_manager.go:414] "containerMap: removing container" podUID="019be6f3-aa64-4bee-955c-31bb2e24af98" containerName="cleanup"
Oct 30 17:00:37.411909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.411696    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="019be6f3-aa64-4bee-955c-31bb2e24af98" containerName="cleanup"
Oct 30 17:00:37.411909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.411770    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="019be6f3-aa64-4bee-955c-31bb2e24af98" containerName="cleanup"
Oct 30 17:00:37.411909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.411779    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="f41b1834-3bc9-467c-8fff-b7d2558b6271" containerName="cleanup"
Oct 30 17:00:37.411909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.411787    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="87888b19-88eb-4c0b-96fd-53bf3f0dedb6" containerName="cleanup"
Oct 30 17:00:37.411909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.411792    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220" containerName="cleanup"
Oct 30 17:00:37.411909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.411796    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="f41b1834-3bc9-467c-8fff-b7d2558b6271" containerName="cleanup"
Oct 30 17:00:37.411909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.411801    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="87888b19-88eb-4c0b-96fd-53bf3f0dedb6" containerName="cleanup"
Oct 30 17:00:37.411909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.411805    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="87888b19-88eb-4c0b-96fd-53bf3f0dedb6" containerName="cleanup"
Oct 30 17:00:37.411909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.411810    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220" containerName="cleanup"
Oct 30 17:00:37.411909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.411816    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad253ce-bd3a-4f0b-ba6c-91ba08e5d220" containerName="cleanup"
Oct 30 17:00:37.411909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.411830    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="019be6f3-aa64-4bee-955c-31bb2e24af98" containerName="cleanup"
Oct 30 17:00:37.416510 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.416488    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:00:37.433730 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.433706    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"]
Oct 30 17:00:37.446329 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.446298    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-04248e\"/\"premium-w1-04248e-kserve-self-signed-certs\""
Oct 30 17:00:37.457256 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.457227    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a8d170a-6410-4e32-837d-8a09cc612c6c-tls-certs\") pod \"premium-w1-04248e-kserve-7dcd547c46-g9hv5\" (UID: \"5a8d170a-6410-4e32-837d-8a09cc612c6c\") " pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:00:37.457445 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.457298    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88wcc\" (UniqueName: \"kubernetes.io/projected/5a8d170a-6410-4e32-837d-8a09cc612c6c-kube-api-access-88wcc\") pod \"premium-w1-04248e-kserve-7dcd547c46-g9hv5\" (UID: \"5a8d170a-6410-4e32-837d-8a09cc612c6c\") " pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:00:37.457524 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.457442    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a8d170a-6410-4e32-837d-8a09cc612c6c-home\") pod \"premium-w1-04248e-kserve-7dcd547c46-g9hv5\" (UID: \"5a8d170a-6410-4e32-837d-8a09cc612c6c\") " pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:00:37.457588 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.457545    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a8d170a-6410-4e32-837d-8a09cc612c6c-tmp-dir\") pod \"premium-w1-04248e-kserve-7dcd547c46-g9hv5\" (UID: \"5a8d170a-6410-4e32-837d-8a09cc612c6c\") " pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:00:37.457588 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.457580    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a8d170a-6410-4e32-837d-8a09cc612c6c-dshm\") pod \"premium-w1-04248e-kserve-7dcd547c46-g9hv5\" (UID: \"5a8d170a-6410-4e32-837d-8a09cc612c6c\") " pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:00:37.457698 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.457605    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a8d170a-6410-4e32-837d-8a09cc612c6c-model-cache\") pod \"premium-w1-04248e-kserve-7dcd547c46-g9hv5\" (UID: \"5a8d170a-6410-4e32-837d-8a09cc612c6c\") " pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:00:37.558258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.558207    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a8d170a-6410-4e32-837d-8a09cc612c6c-tls-certs\") pod \"premium-w1-04248e-kserve-7dcd547c46-g9hv5\" (UID: \"5a8d170a-6410-4e32-837d-8a09cc612c6c\") " pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:00:37.558493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.558290    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88wcc\" (UniqueName: \"kubernetes.io/projected/5a8d170a-6410-4e32-837d-8a09cc612c6c-kube-api-access-88wcc\") pod \"premium-w1-04248e-kserve-7dcd547c46-g9hv5\" (UID: \"5a8d170a-6410-4e32-837d-8a09cc612c6c\") " pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:00:37.558493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.558338    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a8d170a-6410-4e32-837d-8a09cc612c6c-home\") pod \"premium-w1-04248e-kserve-7dcd547c46-g9hv5\" (UID: \"5a8d170a-6410-4e32-837d-8a09cc612c6c\") " pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:00:37.558493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.558384    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a8d170a-6410-4e32-837d-8a09cc612c6c-tmp-dir\") pod \"premium-w1-04248e-kserve-7dcd547c46-g9hv5\" (UID: \"5a8d170a-6410-4e32-837d-8a09cc612c6c\") " pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:00:37.558493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.558439    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a8d170a-6410-4e32-837d-8a09cc612c6c-dshm\") pod \"premium-w1-04248e-kserve-7dcd547c46-g9hv5\" (UID: \"5a8d170a-6410-4e32-837d-8a09cc612c6c\") " pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:00:37.558493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.558461    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a8d170a-6410-4e32-837d-8a09cc612c6c-model-cache\") pod \"premium-w1-04248e-kserve-7dcd547c46-g9hv5\" (UID: \"5a8d170a-6410-4e32-837d-8a09cc612c6c\") " pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:00:37.558870 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.558847    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a8d170a-6410-4e32-837d-8a09cc612c6c-model-cache\") pod \"premium-w1-04248e-kserve-7dcd547c46-g9hv5\" (UID: \"5a8d170a-6410-4e32-837d-8a09cc612c6c\") " pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:00:37.558870 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.558863    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a8d170a-6410-4e32-837d-8a09cc612c6c-home\") pod \"premium-w1-04248e-kserve-7dcd547c46-g9hv5\" (UID: \"5a8d170a-6410-4e32-837d-8a09cc612c6c\") " pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:00:37.559188 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.559163    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a8d170a-6410-4e32-837d-8a09cc612c6c-tmp-dir\") pod \"premium-w1-04248e-kserve-7dcd547c46-g9hv5\" (UID: \"5a8d170a-6410-4e32-837d-8a09cc612c6c\") " pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:00:37.560857 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.560827    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a8d170a-6410-4e32-837d-8a09cc612c6c-dshm\") pod \"premium-w1-04248e-kserve-7dcd547c46-g9hv5\" (UID: \"5a8d170a-6410-4e32-837d-8a09cc612c6c\") " pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:00:37.561086 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.561062    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a8d170a-6410-4e32-837d-8a09cc612c6c-tls-certs\") pod \"premium-w1-04248e-kserve-7dcd547c46-g9hv5\" (UID: \"5a8d170a-6410-4e32-837d-8a09cc612c6c\") " pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:00:37.586684 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.586651    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88wcc\" (UniqueName: \"kubernetes.io/projected/5a8d170a-6410-4e32-837d-8a09cc612c6c-kube-api-access-88wcc\") pod \"premium-w1-04248e-kserve-7dcd547c46-g9hv5\" (UID: \"5a8d170a-6410-4e32-837d-8a09cc612c6c\") " pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:00:37.728109 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:37.728069    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:00:38.029058 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:38.028958    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"]
Oct 30 17:00:38.032140 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:00:38.032098    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a8d170a_6410_4e32_837d_8a09cc612c6c.slice/crio-0c4f7b90b8f7210a398cc8857629b8971487810fd6ab5425fe5edf1925d26c1f WatchSource:0}: Error finding container 0c4f7b90b8f7210a398cc8857629b8971487810fd6ab5425fe5edf1925d26c1f: Status 404 returned error can't find the container with id 0c4f7b90b8f7210a398cc8857629b8971487810fd6ab5425fe5edf1925d26c1f
Oct 30 17:00:38.497128 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:38.497081    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" event={"ID":"5a8d170a-6410-4e32-837d-8a09cc612c6c","Type":"ContainerStarted","Data":"f2af00ad3cdc877e1f0f327f66c6d10fe472bda51a28d38c7e419df07872d902"}
Oct 30 17:00:38.497563 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:38.497139    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" event={"ID":"5a8d170a-6410-4e32-837d-8a09cc612c6c","Type":"ContainerStarted","Data":"0c4f7b90b8f7210a398cc8857629b8971487810fd6ab5425fe5edf1925d26c1f"}
Oct 30 17:00:38.539620 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:38.539558    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podStartSLOduration=1.539543978 podStartE2EDuration="1.539543978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:38.539125035 +0000 UTC m=+814.974027261" watchObservedRunningTime="2026-10-30 17:00:38.539543978 +0000 UTC m=+814.974446184"
Oct 30 17:00:38.729219 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:38.729176    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:00:47.728857 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:47.728814    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:00:47.741579 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:47.741551    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:00:48.558151 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:48.558124    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:00:52.108295 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.108261    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"]
Oct 30 17:00:52.108727 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.108706    4433 cpu_manager.go:414] "containerMap: removing container" podUID="f41b1834-3bc9-467c-8fff-b7d2558b6271" containerName="cleanup"
Oct 30 17:00:52.108727 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.108718    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f41b1834-3bc9-467c-8fff-b7d2558b6271" containerName="cleanup"
Oct 30 17:00:52.108799 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.108741    4433 cpu_manager.go:414] "containerMap: removing container" podUID="019be6f3-aa64-4bee-955c-31bb2e24af98" containerName="cleanup"
Oct 30 17:00:52.108799 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.108746    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="019be6f3-aa64-4bee-955c-31bb2e24af98" containerName="cleanup"
Oct 30 17:00:52.108868 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.108815    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="019be6f3-aa64-4bee-955c-31bb2e24af98" containerName="cleanup"
Oct 30 17:00:52.108868 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.108827    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="f41b1834-3bc9-467c-8fff-b7d2558b6271" containerName="cleanup"
Oct 30 17:00:52.111831 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.111808    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"
Oct 30 17:00:52.139283 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.139247    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"]
Oct 30 17:00:52.213156 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.213124    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dded80b-9e71-4566-944c-4d548c59a684-istiod-ca-cert\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"
Oct 30 17:00:52.213340 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.213195    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dded80b-9e71-4566-944c-4d548c59a684-istio-token\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"
Oct 30 17:00:52.213340 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.213224    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dded80b-9e71-4566-944c-4d548c59a684-workload-certs\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"
Oct 30 17:00:52.213340 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.213283    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvmft\" (UniqueName: \"kubernetes.io/projected/7dded80b-9e71-4566-944c-4d548c59a684-kube-api-access-rvmft\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"
Oct 30 17:00:52.213340 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.213334    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dded80b-9e71-4566-944c-4d548c59a684-workload-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"
Oct 30 17:00:52.213563 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.213434    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dded80b-9e71-4566-944c-4d548c59a684-istio-podinfo\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"
Oct 30 17:00:52.213563 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.213477    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dded80b-9e71-4566-944c-4d548c59a684-credential-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"
Oct 30 17:00:52.213563 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.213493    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dded80b-9e71-4566-944c-4d548c59a684-istio-data\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"
Oct 30 17:00:52.213666 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.213608    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dded80b-9e71-4566-944c-4d548c59a684-istio-envoy\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"
Oct 30 17:00:52.314362 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.314326    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dded80b-9e71-4566-944c-4d548c59a684-credential-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"
Oct 30 17:00:52.314362 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.314360    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dded80b-9e71-4566-944c-4d548c59a684-istio-data\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"
Oct 30 17:00:52.314603 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.314452    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dded80b-9e71-4566-944c-4d548c59a684-istio-envoy\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"
Oct 30 17:00:52.314603 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.314532    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dded80b-9e71-4566-944c-4d548c59a684-istiod-ca-cert\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"
Oct 30 17:00:52.314603 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.314580    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dded80b-9e71-4566-944c-4d548c59a684-istio-token\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"
Oct 30 17:00:52.314763 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.314618    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dded80b-9e71-4566-944c-4d548c59a684-workload-certs\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"
Oct 30 17:00:52.314763 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.314653    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvmft\" (UniqueName: \"kubernetes.io/projected/7dded80b-9e71-4566-944c-4d548c59a684-kube-api-access-rvmft\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"
Oct 30 17:00:52.314870 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.314833    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dded80b-9e71-4566-944c-4d548c59a684-credential-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"
Oct 30 17:00:52.314870 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.314843    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dded80b-9e71-4566-944c-4d548c59a684-istio-data\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"
Oct 30 17:00:52.314982 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.314874    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dded80b-9e71-4566-944c-4d548c59a684-workload-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"
Oct 30 17:00:52.314982 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.314966    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dded80b-9e71-4566-944c-4d548c59a684-istio-podinfo\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"
Oct 30 17:00:52.315142 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.315118    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dded80b-9e71-4566-944c-4d548c59a684-workload-certs\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"
Oct 30 17:00:52.315223 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.315205    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dded80b-9e71-4566-944c-4d548c59a684-workload-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"
Oct 30 17:00:52.315340 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.315318    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dded80b-9e71-4566-944c-4d548c59a684-istiod-ca-cert\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"
Oct 30 17:00:52.316805 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.316780    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dded80b-9e71-4566-944c-4d548c59a684-istio-envoy\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"
Oct 30 17:00:52.317295 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.317273    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dded80b-9e71-4566-944c-4d548c59a684-istio-podinfo\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"
Oct 30 17:00:52.374676 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.374595    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dded80b-9e71-4566-944c-4d548c59a684-istio-token\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"
Oct 30 17:00:52.374818 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.374693    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvmft\" (UniqueName: \"kubernetes.io/projected/7dded80b-9e71-4566-944c-4d548c59a684-kube-api-access-rvmft\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"
Oct 30 17:00:52.428646 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.428610    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"
Oct 30 17:00:52.752256 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.752225    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"]
Oct 30 17:00:52.753846 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:00:52.753817    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dded80b_9e71_4566_944c_4d548c59a684.slice/crio-18210d37ee2d6df7460d2fabaacbe0a7ab1b0846ccbe693ef2fda383d5a1ad2f WatchSource:0}: Error finding container 18210d37ee2d6df7460d2fabaacbe0a7ab1b0846ccbe693ef2fda383d5a1ad2f: Status 404 returned error can't find the container with id 18210d37ee2d6df7460d2fabaacbe0a7ab1b0846ccbe693ef2fda383d5a1ad2f
Oct 30 17:00:52.756247 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.756216    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:00:52.756317 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.756281    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:00:52.756317 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:52.756310    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:00:53.577744 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:53.577694    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz" event={"ID":"7dded80b-9e71-4566-944c-4d548c59a684","Type":"ContainerStarted","Data":"13e25913a9461eb4e01fabbd28fa4000ee85fb8b7c9d7ed13b15baa9f97396eb"}
Oct 30 17:00:53.579775 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:53.579746    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz" event={"ID":"7dded80b-9e71-4566-944c-4d548c59a684","Type":"ContainerStarted","Data":"18210d37ee2d6df7460d2fabaacbe0a7ab1b0846ccbe693ef2fda383d5a1ad2f"}
Oct 30 17:00:53.642875 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:53.642802    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz" podStartSLOduration=1.642788151 podStartE2EDuration="1.642788151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:53.640082743 +0000 UTC m=+830.074984948" watchObservedRunningTime="2026-10-30 17:00:53.642788151 +0000 UTC m=+830.077690354"
Oct 30 17:00:54.429153 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:54.429114    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"
Oct 30 17:00:54.430842 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:54.430816    4433 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 17:00:54.430966 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:54.430877    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz" podUID="7dded80b-9e71-4566-944c-4d548c59a684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 17:00:55.429754 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:55.429708    4433 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 17:00:55.430179 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:55.429779    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz" podUID="7dded80b-9e71-4566-944c-4d548c59a684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 17:00:56.429793 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:56.429762    4433 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 17:00:56.430170 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:56.429810    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz" podUID="7dded80b-9e71-4566-944c-4d548c59a684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 17:00:57.002733 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:00:57.002696    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:00:57.429858 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:57.429823    4433 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 17:00:57.430223 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:57.429886    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz" podUID="7dded80b-9e71-4566-944c-4d548c59a684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 17:00:58.429274 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:58.429229    4433 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 17:00:58.429509 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:58.429292    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz" podUID="7dded80b-9e71-4566-944c-4d548c59a684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 17:00:59.429787 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:59.429737    4433 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 17:00:59.430242 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:00:59.429805    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz" podUID="7dded80b-9e71-4566-944c-4d548c59a684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 17:01:00.433866 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:00.433832    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"
Oct 30 17:01:00.434311 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:00.434242    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"
Oct 30 17:01:00.435016 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:00.435001    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"
Oct 30 17:01:04.779131 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:04.779094    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:01:07.034075 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.034032    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"]
Oct 30 17:01:07.040098 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.040076    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"
Oct 30 17:01:07.074727 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.074685    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-c7b9df-openshift-default-dockercfg-qhctr\""
Oct 30 17:01:07.077090 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.077062    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"]
Oct 30 17:01:07.123584 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.123545    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee372e3-661e-49b0-ae79-6c7ae7736955-workload-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"
Oct 30 17:01:07.123774 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.123688    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ee372e3-661e-49b0-ae79-6c7ae7736955-istio-envoy\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"
Oct 30 17:01:07.123774 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.123752    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ee372e3-661e-49b0-ae79-6c7ae7736955-istiod-ca-cert\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"
Oct 30 17:01:07.123880 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.123844    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee372e3-661e-49b0-ae79-6c7ae7736955-credential-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"
Oct 30 17:01:07.123931 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.123883    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ee372e3-661e-49b0-ae79-6c7ae7736955-workload-certs\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"
Oct 30 17:01:07.123999 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.123983    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgxzv\" (UniqueName: \"kubernetes.io/projected/1ee372e3-661e-49b0-ae79-6c7ae7736955-kube-api-access-bgxzv\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"
Oct 30 17:01:07.124039 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.124026    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ee372e3-661e-49b0-ae79-6c7ae7736955-istio-data\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"
Oct 30 17:01:07.124121 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.124107    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ee372e3-661e-49b0-ae79-6c7ae7736955-istio-podinfo\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"
Oct 30 17:01:07.124192 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.124171    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ee372e3-661e-49b0-ae79-6c7ae7736955-istio-token\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"
Oct 30 17:01:07.225706 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.225666    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee372e3-661e-49b0-ae79-6c7ae7736955-workload-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"
Oct 30 17:01:07.225902 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.225731    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ee372e3-661e-49b0-ae79-6c7ae7736955-istio-envoy\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"
Oct 30 17:01:07.225902 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.225760    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ee372e3-661e-49b0-ae79-6c7ae7736955-istiod-ca-cert\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"
Oct 30 17:01:07.225902 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.225781    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee372e3-661e-49b0-ae79-6c7ae7736955-credential-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"
Oct 30 17:01:07.225902 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.225796    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ee372e3-661e-49b0-ae79-6c7ae7736955-workload-certs\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"
Oct 30 17:01:07.225902 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.225836    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgxzv\" (UniqueName: \"kubernetes.io/projected/1ee372e3-661e-49b0-ae79-6c7ae7736955-kube-api-access-bgxzv\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"
Oct 30 17:01:07.225902 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.225876    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ee372e3-661e-49b0-ae79-6c7ae7736955-istio-data\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"
Oct 30 17:01:07.226235 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.225937    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ee372e3-661e-49b0-ae79-6c7ae7736955-istio-podinfo\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"
Oct 30 17:01:07.226235 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.225968    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ee372e3-661e-49b0-ae79-6c7ae7736955-istio-token\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"
Oct 30 17:01:07.226235 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.226128    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee372e3-661e-49b0-ae79-6c7ae7736955-workload-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"
Oct 30 17:01:07.226235 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.226137    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee372e3-661e-49b0-ae79-6c7ae7736955-credential-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"
Oct 30 17:01:07.226523 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.226495    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ee372e3-661e-49b0-ae79-6c7ae7736955-istio-data\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"
Oct 30 17:01:07.226590 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.226565    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ee372e3-661e-49b0-ae79-6c7ae7736955-workload-certs\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"
Oct 30 17:01:07.226652 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.226610    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ee372e3-661e-49b0-ae79-6c7ae7736955-istiod-ca-cert\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"
Oct 30 17:01:07.229018 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.228989    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ee372e3-661e-49b0-ae79-6c7ae7736955-istio-envoy\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"
Oct 30 17:01:07.229131 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.229112    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ee372e3-661e-49b0-ae79-6c7ae7736955-istio-podinfo\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"
Oct 30 17:01:07.253552 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.253521    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ee372e3-661e-49b0-ae79-6c7ae7736955-istio-token\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"
Oct 30 17:01:07.253728 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.253670    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgxzv\" (UniqueName: \"kubernetes.io/projected/1ee372e3-661e-49b0-ae79-6c7ae7736955-kube-api-access-bgxzv\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"
Oct 30 17:01:07.354836 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.354738    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"
Oct 30 17:01:07.677320 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.677285    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"]
Oct 30 17:01:07.680293 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:01:07.680246    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ee372e3_661e_49b0_ae79_6c7ae7736955.slice/crio-90bc589bcb024c72592857bf72487ac00008930c939447e4e8b33290a03a976f WatchSource:0}: Error finding container 90bc589bcb024c72592857bf72487ac00008930c939447e4e8b33290a03a976f: Status 404 returned error can't find the container with id 90bc589bcb024c72592857bf72487ac00008930c939447e4e8b33290a03a976f
Oct 30 17:01:07.683147 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.683111    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:01:07.683264 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.683185    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:01:07.683264 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:07.683213    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:01:08.681473 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:08.681437    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr" event={"ID":"1ee372e3-661e-49b0-ae79-6c7ae7736955","Type":"ContainerStarted","Data":"467873d1735e34e1aafdc99c7f54ef4e8fe7e4982229d10050932dd70f3b808f"}
Oct 30 17:01:08.681994 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:08.681507    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr" event={"ID":"1ee372e3-661e-49b0-ae79-6c7ae7736955","Type":"ContainerStarted","Data":"90bc589bcb024c72592857bf72487ac00008930c939447e4e8b33290a03a976f"}
Oct 30 17:01:08.740308 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:08.740255    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr" podStartSLOduration=2.740239478 podStartE2EDuration="2.740239478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:08.737292873 +0000 UTC m=+845.172195080" watchObservedRunningTime="2026-10-30 17:01:08.740239478 +0000 UTC m=+845.175141684"
Oct 30 17:01:09.358427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:09.357368    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"
Oct 30 17:01:09.358427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:09.357553    4433 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 17:01:09.358427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:09.357598    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr" podUID="1ee372e3-661e-49b0-ae79-6c7ae7736955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 17:01:10.374429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:10.370616    4433 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 17:01:10.374429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:10.370677    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr" podUID="1ee372e3-661e-49b0-ae79-6c7ae7736955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 17:01:11.358429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:11.357799    4433 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 17:01:11.358429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:11.357862    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr" podUID="1ee372e3-661e-49b0-ae79-6c7ae7736955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 17:01:12.357973 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:12.357935    4433 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 17:01:12.358630 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:12.358597    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr" podUID="1ee372e3-661e-49b0-ae79-6c7ae7736955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 17:01:13.016219 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:13.016185    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e9671d-xmwqb"]
Oct 30 17:01:13.020120 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:13.020103    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e9671d-xmwqb"
Oct 30 17:01:13.036594 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:13.036571    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:01:13.048166 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:13.048142    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\""
Oct 30 17:01:13.076428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:13.074448    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e9671d-xmwqb"]
Oct 30 17:01:13.109243 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:13.109213    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5h95\" (UniqueName: \"kubernetes.io/projected/3e098fdc-ae03-4286-9891-cd5a81343358-kube-api-access-w5h95\") pod \"maas-api-revoke-sub-28e9671d-xmwqb\" (UID: \"3e098fdc-ae03-4286-9891-cd5a81343358\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e9671d-xmwqb"
Oct 30 17:01:13.109442 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:13.109266    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e098fdc-ae03-4286-9891-cd5a81343358-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28e9671d-xmwqb\" (UID: \"3e098fdc-ae03-4286-9891-cd5a81343358\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e9671d-xmwqb"
Oct 30 17:01:13.109442 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:13.109314    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e098fdc-ae03-4286-9891-cd5a81343358-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28e9671d-xmwqb\" (UID: \"3e098fdc-ae03-4286-9891-cd5a81343358\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e9671d-xmwqb"
Oct 30 17:01:13.210538 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:13.210493    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5h95\" (UniqueName: \"kubernetes.io/projected/3e098fdc-ae03-4286-9891-cd5a81343358-kube-api-access-w5h95\") pod \"maas-api-revoke-sub-28e9671d-xmwqb\" (UID: \"3e098fdc-ae03-4286-9891-cd5a81343358\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e9671d-xmwqb"
Oct 30 17:01:13.210746 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:13.210553    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e098fdc-ae03-4286-9891-cd5a81343358-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28e9671d-xmwqb\" (UID: \"3e098fdc-ae03-4286-9891-cd5a81343358\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e9671d-xmwqb"
Oct 30 17:01:13.210746 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:13.210595    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e098fdc-ae03-4286-9891-cd5a81343358-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28e9671d-xmwqb\" (UID: \"3e098fdc-ae03-4286-9891-cd5a81343358\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e9671d-xmwqb"
Oct 30 17:01:13.211202 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:13.211179    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e098fdc-ae03-4286-9891-cd5a81343358-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28e9671d-xmwqb\" (UID: \"3e098fdc-ae03-4286-9891-cd5a81343358\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e9671d-xmwqb"
Oct 30 17:01:13.213077 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:13.213051    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e098fdc-ae03-4286-9891-cd5a81343358-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28e9671d-xmwqb\" (UID: \"3e098fdc-ae03-4286-9891-cd5a81343358\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e9671d-xmwqb"
Oct 30 17:01:13.233978 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:13.233954    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5h95\" (UniqueName: \"kubernetes.io/projected/3e098fdc-ae03-4286-9891-cd5a81343358-kube-api-access-w5h95\") pod \"maas-api-revoke-sub-28e9671d-xmwqb\" (UID: \"3e098fdc-ae03-4286-9891-cd5a81343358\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e9671d-xmwqb"
Oct 30 17:01:13.374566 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:13.366842    4433 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 17:01:13.374566 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:13.366900    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr" podUID="1ee372e3-661e-49b0-ae79-6c7ae7736955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 17:01:13.388749 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:13.388666    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e9671d-xmwqb"
Oct 30 17:01:13.696596 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:13.696567    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e9671d-xmwqb"]
Oct 30 17:01:13.697434 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:01:13.697373    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e098fdc_ae03_4286_9891_cd5a81343358.slice/crio-8a5ee57a49a0a0da995bc038c3b73e65da6b7c0b6625a8325d7666e998c56f35 WatchSource:0}: Error finding container 8a5ee57a49a0a0da995bc038c3b73e65da6b7c0b6625a8325d7666e998c56f35: Status 404 returned error can't find the container with id 8a5ee57a49a0a0da995bc038c3b73e65da6b7c0b6625a8325d7666e998c56f35
Oct 30 17:01:13.706462 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:13.706435    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e9671d-xmwqb" event={"ID":"3e098fdc-ae03-4286-9891-cd5a81343358","Type":"ContainerStarted","Data":"8a5ee57a49a0a0da995bc038c3b73e65da6b7c0b6625a8325d7666e998c56f35"}
Oct 30 17:01:14.356046 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:14.355963    4433 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 17:01:14.356046 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:14.356025    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr" podUID="1ee372e3-661e-49b0-ae79-6c7ae7736955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 17:01:14.712632 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:14.712599    4433 generic.go:356] "Generic (PLEG): container finished" podID="3e098fdc-ae03-4286-9891-cd5a81343358" containerID="6420f1bff3dbc7c2b261cb24dd8a07bfcf4adf7717747a73b288d7c0bd41fd34" exitCode=0
Oct 30 17:01:14.713047 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:14.712685    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e9671d-xmwqb" event={"ID":"3e098fdc-ae03-4286-9891-cd5a81343358","Type":"ContainerDied","Data":"6420f1bff3dbc7c2b261cb24dd8a07bfcf4adf7717747a73b288d7c0bd41fd34"}
Oct 30 17:01:15.359693 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:15.359661    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"
Oct 30 17:01:15.359968 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:15.359947    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"
Oct 30 17:01:15.360729 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:15.360710    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"
Oct 30 17:01:15.593058 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:15.593026    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1cfa206-5m555"]
Oct 30 17:01:15.596861 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:15.596845    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cfa206-5m555"
Oct 30 17:01:15.621138 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:15.621073    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1cfa206-5m555"]
Oct 30 17:01:15.637988 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:15.637954    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8edded39-42a8-4d0b-bfb1-d5b77d3102fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1cfa206-5m555\" (UID: \"8edded39-42a8-4d0b-bfb1-d5b77d3102fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cfa206-5m555"
Oct 30 17:01:15.638142 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:15.638004    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxflw\" (UniqueName: \"kubernetes.io/projected/8edded39-42a8-4d0b-bfb1-d5b77d3102fd-kube-api-access-hxflw\") pod \"maas-api-revoke-sub-f1cfa206-5m555\" (UID: \"8edded39-42a8-4d0b-bfb1-d5b77d3102fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cfa206-5m555"
Oct 30 17:01:15.638234 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:15.638219    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8edded39-42a8-4d0b-bfb1-d5b77d3102fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1cfa206-5m555\" (UID: \"8edded39-42a8-4d0b-bfb1-d5b77d3102fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cfa206-5m555"
Oct 30 17:01:15.738802 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:15.738770    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8edded39-42a8-4d0b-bfb1-d5b77d3102fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1cfa206-5m555\" (UID: \"8edded39-42a8-4d0b-bfb1-d5b77d3102fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cfa206-5m555"
Oct 30 17:01:15.739178 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:15.738809    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxflw\" (UniqueName: \"kubernetes.io/projected/8edded39-42a8-4d0b-bfb1-d5b77d3102fd-kube-api-access-hxflw\") pod \"maas-api-revoke-sub-f1cfa206-5m555\" (UID: \"8edded39-42a8-4d0b-bfb1-d5b77d3102fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cfa206-5m555"
Oct 30 17:01:15.739178 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:15.738927    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8edded39-42a8-4d0b-bfb1-d5b77d3102fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1cfa206-5m555\" (UID: \"8edded39-42a8-4d0b-bfb1-d5b77d3102fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cfa206-5m555"
Oct 30 17:01:15.739778 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:15.739752    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8edded39-42a8-4d0b-bfb1-d5b77d3102fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1cfa206-5m555\" (UID: \"8edded39-42a8-4d0b-bfb1-d5b77d3102fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cfa206-5m555"
Oct 30 17:01:15.741191 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:15.741165    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8edded39-42a8-4d0b-bfb1-d5b77d3102fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1cfa206-5m555\" (UID: \"8edded39-42a8-4d0b-bfb1-d5b77d3102fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cfa206-5m555"
Oct 30 17:01:15.762933 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:15.762906    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxflw\" (UniqueName: \"kubernetes.io/projected/8edded39-42a8-4d0b-bfb1-d5b77d3102fd-kube-api-access-hxflw\") pod \"maas-api-revoke-sub-f1cfa206-5m555\" (UID: \"8edded39-42a8-4d0b-bfb1-d5b77d3102fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cfa206-5m555"
Oct 30 17:01:15.848702 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:15.848674    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e9671d-xmwqb"
Oct 30 17:01:15.909371 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:15.909250    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cfa206-5m555"
Oct 30 17:01:15.941601 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:15.941570    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e098fdc-ae03-4286-9891-cd5a81343358-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e098fdc-ae03-4286-9891-cd5a81343358-maas-api-xks-ca\") pod \"3e098fdc-ae03-4286-9891-cd5a81343358\" (UID: \"3e098fdc-ae03-4286-9891-cd5a81343358\") "
Oct 30 17:01:15.941749 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:15.941634    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e098fdc-ae03-4286-9891-cd5a81343358-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e098fdc-ae03-4286-9891-cd5a81343358-maas-api-openshift-ca\") pod \"3e098fdc-ae03-4286-9891-cd5a81343358\" (UID: \"3e098fdc-ae03-4286-9891-cd5a81343358\") "
Oct 30 17:01:15.941799 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:15.941759    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e098fdc-ae03-4286-9891-cd5a81343358-kube-api-access-w5h95\" (UniqueName: \"kubernetes.io/projected/3e098fdc-ae03-4286-9891-cd5a81343358-kube-api-access-w5h95\") pod \"3e098fdc-ae03-4286-9891-cd5a81343358\" (UID: \"3e098fdc-ae03-4286-9891-cd5a81343358\") "
Oct 30 17:01:15.942010 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:15.941982    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e098fdc-ae03-4286-9891-cd5a81343358-maas-api-openshift-ca" pod "3e098fdc-ae03-4286-9891-cd5a81343358" (UID: "3e098fdc-ae03-4286-9891-cd5a81343358"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:01:15.942218 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:15.942201    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e098fdc-ae03-4286-9891-cd5a81343358-maas-api-openshift-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:01:15.943697 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:15.943670    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e098fdc-ae03-4286-9891-cd5a81343358-maas-api-xks-ca" pod "3e098fdc-ae03-4286-9891-cd5a81343358" (UID: "3e098fdc-ae03-4286-9891-cd5a81343358"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:01:15.943848 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:15.943832    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e098fdc-ae03-4286-9891-cd5a81343358-kube-api-access-w5h95" pod "3e098fdc-ae03-4286-9891-cd5a81343358" (UID: "3e098fdc-ae03-4286-9891-cd5a81343358"). InnerVolumeSpecName "kube-api-access-w5h95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:01:16.043027 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:16.042990    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e098fdc-ae03-4286-9891-cd5a81343358-maas-api-xks-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:01:16.043027 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:16.043020    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5h95\" (UniqueName: \"kubernetes.io/projected/3e098fdc-ae03-4286-9891-cd5a81343358-kube-api-access-w5h95\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:01:16.060183 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:16.060144    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b089650-8x8fb"]
Oct 30 17:01:16.060716 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:16.060687    4433 cpu_manager.go:414] "containerMap: removing container" podUID="3e098fdc-ae03-4286-9891-cd5a81343358" containerName="revoke-keys"
Oct 30 17:01:16.060716 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:16.060710    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e098fdc-ae03-4286-9891-cd5a81343358" containerName="revoke-keys"
Oct 30 17:01:16.060914 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:16.060882    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e098fdc-ae03-4286-9891-cd5a81343358" containerName="revoke-keys"
Oct 30 17:01:16.069567 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:16.069541    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b089650-8x8fb"
Oct 30 17:01:16.084285 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:16.084257    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b089650-8x8fb"]
Oct 30 17:01:16.144599 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:16.144560    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qlnv\" (UniqueName: \"kubernetes.io/projected/08dd97a0-74fd-4b66-84ff-47717c5b8849-kube-api-access-9qlnv\") pod \"maas-api-revoke-sub-5b089650-8x8fb\" (UID: \"08dd97a0-74fd-4b66-84ff-47717c5b8849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b089650-8x8fb"
Oct 30 17:01:16.144779 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:16.144672    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08dd97a0-74fd-4b66-84ff-47717c5b8849-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b089650-8x8fb\" (UID: \"08dd97a0-74fd-4b66-84ff-47717c5b8849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b089650-8x8fb"
Oct 30 17:01:16.144779 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:16.144759    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08dd97a0-74fd-4b66-84ff-47717c5b8849-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b089650-8x8fb\" (UID: \"08dd97a0-74fd-4b66-84ff-47717c5b8849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b089650-8x8fb"
Oct 30 17:01:16.209693 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:16.209653    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1cfa206-5m555"]
Oct 30 17:01:16.214016 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:01:16.213993    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8edded39_42a8_4d0b_bfb1_d5b77d3102fd.slice/crio-87e30379b6eb0e82ed8216239d70527b648987aa4b8f7673ba71d1b75943ab6a WatchSource:0}: Error finding container 87e30379b6eb0e82ed8216239d70527b648987aa4b8f7673ba71d1b75943ab6a: Status 404 returned error can't find the container with id 87e30379b6eb0e82ed8216239d70527b648987aa4b8f7673ba71d1b75943ab6a
Oct 30 17:01:16.245531 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:16.245508    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08dd97a0-74fd-4b66-84ff-47717c5b8849-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b089650-8x8fb\" (UID: \"08dd97a0-74fd-4b66-84ff-47717c5b8849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b089650-8x8fb"
Oct 30 17:01:16.245629 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:16.245546    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08dd97a0-74fd-4b66-84ff-47717c5b8849-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b089650-8x8fb\" (UID: \"08dd97a0-74fd-4b66-84ff-47717c5b8849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b089650-8x8fb"
Oct 30 17:01:16.245669 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:16.245663    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qlnv\" (UniqueName: \"kubernetes.io/projected/08dd97a0-74fd-4b66-84ff-47717c5b8849-kube-api-access-9qlnv\") pod \"maas-api-revoke-sub-5b089650-8x8fb\" (UID: \"08dd97a0-74fd-4b66-84ff-47717c5b8849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b089650-8x8fb"
Oct 30 17:01:16.246249 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:16.246226    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08dd97a0-74fd-4b66-84ff-47717c5b8849-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b089650-8x8fb\" (UID: \"08dd97a0-74fd-4b66-84ff-47717c5b8849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b089650-8x8fb"
Oct 30 17:01:16.247806 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:16.247787    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08dd97a0-74fd-4b66-84ff-47717c5b8849-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b089650-8x8fb\" (UID: \"08dd97a0-74fd-4b66-84ff-47717c5b8849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b089650-8x8fb"
Oct 30 17:01:16.269993 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:16.269967    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qlnv\" (UniqueName: \"kubernetes.io/projected/08dd97a0-74fd-4b66-84ff-47717c5b8849-kube-api-access-9qlnv\") pod \"maas-api-revoke-sub-5b089650-8x8fb\" (UID: \"08dd97a0-74fd-4b66-84ff-47717c5b8849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b089650-8x8fb"
Oct 30 17:01:16.383190 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:16.383161    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b089650-8x8fb"
Oct 30 17:01:16.683096 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:16.683061    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b089650-8x8fb"]
Oct 30 17:01:16.689557 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:01:16.689530    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08dd97a0_74fd_4b66_84ff_47717c5b8849.slice/crio-6fc014244bb6b2f8e2eb5b9c77754edfc727896c045910a8c2137f331e568530 WatchSource:0}: Error finding container 6fc014244bb6b2f8e2eb5b9c77754edfc727896c045910a8c2137f331e568530: Status 404 returned error can't find the container with id 6fc014244bb6b2f8e2eb5b9c77754edfc727896c045910a8c2137f331e568530
Oct 30 17:01:16.723691 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:16.723671    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e9671d-xmwqb"
Oct 30 17:01:16.723788 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:16.723668    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e9671d-xmwqb" event={"ID":"3e098fdc-ae03-4286-9891-cd5a81343358","Type":"ContainerDied","Data":"8a5ee57a49a0a0da995bc038c3b73e65da6b7c0b6625a8325d7666e998c56f35"}
Oct 30 17:01:16.723848 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:16.723799    4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a5ee57a49a0a0da995bc038c3b73e65da6b7c0b6625a8325d7666e998c56f35"
Oct 30 17:01:16.724960 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:16.724938    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b089650-8x8fb" event={"ID":"08dd97a0-74fd-4b66-84ff-47717c5b8849","Type":"ContainerStarted","Data":"6fc014244bb6b2f8e2eb5b9c77754edfc727896c045910a8c2137f331e568530"}
Oct 30 17:01:16.726533 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:16.726503    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cfa206-5m555" event={"ID":"8edded39-42a8-4d0b-bfb1-d5b77d3102fd","Type":"ContainerStarted","Data":"7b380428a0e3c1a487c07b65ca00e74224f6f410daba9dd31da7c650503ca23a"}
Oct 30 17:01:16.726637 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:16.726549    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cfa206-5m555" event={"ID":"8edded39-42a8-4d0b-bfb1-d5b77d3102fd","Type":"ContainerStarted","Data":"87e30379b6eb0e82ed8216239d70527b648987aa4b8f7673ba71d1b75943ab6a"}
Oct 30 17:01:16.766738 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:16.766690    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cfa206-5m555" podStartSLOduration=1.766676233 podStartE2EDuration="1.766676233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:16.766057197 +0000 UTC m=+853.200959413" watchObservedRunningTime="2026-10-30 17:01:16.766676233 +0000 UTC m=+853.201578439"
Oct 30 17:01:17.734193 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:17.734156    4433 generic.go:356] "Generic (PLEG): container finished" podID="08dd97a0-74fd-4b66-84ff-47717c5b8849" containerID="166c8f0fe5463c968aabb39a7d0200d5420dc37792f11d7a2dedcaed2765fc44" exitCode=0
Oct 30 17:01:17.734378 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:17.734244    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b089650-8x8fb" event={"ID":"08dd97a0-74fd-4b66-84ff-47717c5b8849","Type":"ContainerDied","Data":"166c8f0fe5463c968aabb39a7d0200d5420dc37792f11d7a2dedcaed2765fc44"}
Oct 30 17:01:17.735951 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:17.735920    4433 generic.go:356] "Generic (PLEG): container finished" podID="8edded39-42a8-4d0b-bfb1-d5b77d3102fd" containerID="7b380428a0e3c1a487c07b65ca00e74224f6f410daba9dd31da7c650503ca23a" exitCode=0
Oct 30 17:01:17.736071 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:17.736003    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cfa206-5m555" event={"ID":"8edded39-42a8-4d0b-bfb1-d5b77d3102fd","Type":"ContainerDied","Data":"7b380428a0e3c1a487c07b65ca00e74224f6f410daba9dd31da7c650503ca23a"}
Oct 30 17:01:18.886700 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:18.886665    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cfa206-5m555"
Oct 30 17:01:18.916096 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:18.916070    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b089650-8x8fb"
Oct 30 17:01:18.976465 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:18.976431    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8edded39-42a8-4d0b-bfb1-d5b77d3102fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8edded39-42a8-4d0b-bfb1-d5b77d3102fd-maas-api-openshift-ca\") pod \"8edded39-42a8-4d0b-bfb1-d5b77d3102fd\" (UID: \"8edded39-42a8-4d0b-bfb1-d5b77d3102fd\") "
Oct 30 17:01:18.976650 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:18.976601    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8edded39-42a8-4d0b-bfb1-d5b77d3102fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8edded39-42a8-4d0b-bfb1-d5b77d3102fd-maas-api-xks-ca\") pod \"8edded39-42a8-4d0b-bfb1-d5b77d3102fd\" (UID: \"8edded39-42a8-4d0b-bfb1-d5b77d3102fd\") "
Oct 30 17:01:18.976719 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:18.976653    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08dd97a0-74fd-4b66-84ff-47717c5b8849-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08dd97a0-74fd-4b66-84ff-47717c5b8849-maas-api-xks-ca\") pod \"08dd97a0-74fd-4b66-84ff-47717c5b8849\" (UID: \"08dd97a0-74fd-4b66-84ff-47717c5b8849\") "
Oct 30 17:01:18.976779 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:18.976730    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08dd97a0-74fd-4b66-84ff-47717c5b8849-kube-api-access-9qlnv\" (UniqueName: \"kubernetes.io/projected/08dd97a0-74fd-4b66-84ff-47717c5b8849-kube-api-access-9qlnv\") pod \"08dd97a0-74fd-4b66-84ff-47717c5b8849\" (UID: \"08dd97a0-74fd-4b66-84ff-47717c5b8849\") "
Oct 30 17:01:18.976831 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:18.976792    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08dd97a0-74fd-4b66-84ff-47717c5b8849-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08dd97a0-74fd-4b66-84ff-47717c5b8849-maas-api-openshift-ca\") pod \"08dd97a0-74fd-4b66-84ff-47717c5b8849\" (UID: \"08dd97a0-74fd-4b66-84ff-47717c5b8849\") "
Oct 30 17:01:18.976831 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:18.976820    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8edded39-42a8-4d0b-bfb1-d5b77d3102fd-kube-api-access-hxflw\" (UniqueName: \"kubernetes.io/projected/8edded39-42a8-4d0b-bfb1-d5b77d3102fd-kube-api-access-hxflw\") pod \"8edded39-42a8-4d0b-bfb1-d5b77d3102fd\" (UID: \"8edded39-42a8-4d0b-bfb1-d5b77d3102fd\") "
Oct 30 17:01:18.976934 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:18.976824    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8edded39-42a8-4d0b-bfb1-d5b77d3102fd-maas-api-openshift-ca" pod "8edded39-42a8-4d0b-bfb1-d5b77d3102fd" (UID: "8edded39-42a8-4d0b-bfb1-d5b77d3102fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:01:18.977216 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:18.977180    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08dd97a0-74fd-4b66-84ff-47717c5b8849-maas-api-openshift-ca" pod "08dd97a0-74fd-4b66-84ff-47717c5b8849" (UID: "08dd97a0-74fd-4b66-84ff-47717c5b8849"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:01:18.977339 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:18.977242    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8edded39-42a8-4d0b-bfb1-d5b77d3102fd-maas-api-openshift-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:01:18.978933 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:18.978901    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8edded39-42a8-4d0b-bfb1-d5b77d3102fd-maas-api-xks-ca" pod "8edded39-42a8-4d0b-bfb1-d5b77d3102fd" (UID: "8edded39-42a8-4d0b-bfb1-d5b77d3102fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:01:18.979277 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:18.979252    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08dd97a0-74fd-4b66-84ff-47717c5b8849-kube-api-access-9qlnv" pod "08dd97a0-74fd-4b66-84ff-47717c5b8849" (UID: "08dd97a0-74fd-4b66-84ff-47717c5b8849"). InnerVolumeSpecName "kube-api-access-9qlnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:01:18.979623 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:18.979591    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8edded39-42a8-4d0b-bfb1-d5b77d3102fd-kube-api-access-hxflw" pod "8edded39-42a8-4d0b-bfb1-d5b77d3102fd" (UID: "8edded39-42a8-4d0b-bfb1-d5b77d3102fd"). InnerVolumeSpecName "kube-api-access-hxflw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:01:18.979725 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:18.979651    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08dd97a0-74fd-4b66-84ff-47717c5b8849-maas-api-xks-ca" pod "08dd97a0-74fd-4b66-84ff-47717c5b8849" (UID: "08dd97a0-74fd-4b66-84ff-47717c5b8849"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:01:19.078267 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:19.078182    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08dd97a0-74fd-4b66-84ff-47717c5b8849-maas-api-openshift-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:01:19.078267 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:19.078210    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxflw\" (UniqueName: \"kubernetes.io/projected/8edded39-42a8-4d0b-bfb1-d5b77d3102fd-kube-api-access-hxflw\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:01:19.078267 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:19.078219    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8edded39-42a8-4d0b-bfb1-d5b77d3102fd-maas-api-xks-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:01:19.078267 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:19.078228    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08dd97a0-74fd-4b66-84ff-47717c5b8849-maas-api-xks-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:01:19.078267 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:19.078238    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qlnv\" (UniqueName: \"kubernetes.io/projected/08dd97a0-74fd-4b66-84ff-47717c5b8849-kube-api-access-9qlnv\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:01:19.479903 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:19.479869    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24756d7a-pwrs7"]
Oct 30 17:01:19.480395 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:19.480373    4433 cpu_manager.go:414] "containerMap: removing container" podUID="08dd97a0-74fd-4b66-84ff-47717c5b8849" containerName="revoke-keys"
Oct 30 17:01:19.480395 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:19.480390    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08dd97a0-74fd-4b66-84ff-47717c5b8849" containerName="revoke-keys"
Oct 30 17:01:19.480551 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:19.480449    4433 cpu_manager.go:414] "containerMap: removing container" podUID="8edded39-42a8-4d0b-bfb1-d5b77d3102fd" containerName="revoke-keys"
Oct 30 17:01:19.480551 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:19.480455    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8edded39-42a8-4d0b-bfb1-d5b77d3102fd" containerName="revoke-keys"
Oct 30 17:01:19.480551 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:19.480546    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="8edded39-42a8-4d0b-bfb1-d5b77d3102fd" containerName="revoke-keys"
Oct 30 17:01:19.480670 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:19.480560    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="08dd97a0-74fd-4b66-84ff-47717c5b8849" containerName="revoke-keys"
Oct 30 17:01:19.484234 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:19.484214    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24756d7a-pwrs7"
Oct 30 17:01:19.506529 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:19.506497    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24756d7a-pwrs7"]
Oct 30 17:01:19.583669 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:19.583631    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7c0d558-7283-4304-a4a6-1a9a110ba265-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24756d7a-pwrs7\" (UID: \"d7c0d558-7283-4304-a4a6-1a9a110ba265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24756d7a-pwrs7"
Oct 30 17:01:19.583849 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:19.583694    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7c0d558-7283-4304-a4a6-1a9a110ba265-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24756d7a-pwrs7\" (UID: \"d7c0d558-7283-4304-a4a6-1a9a110ba265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24756d7a-pwrs7"
Oct 30 17:01:19.583849 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:19.583812    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kttzs\" (UniqueName: \"kubernetes.io/projected/d7c0d558-7283-4304-a4a6-1a9a110ba265-kube-api-access-kttzs\") pod \"maas-api-revoke-sub-24756d7a-pwrs7\" (UID: \"d7c0d558-7283-4304-a4a6-1a9a110ba265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24756d7a-pwrs7"
Oct 30 17:01:19.685242 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:19.685189    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7c0d558-7283-4304-a4a6-1a9a110ba265-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24756d7a-pwrs7\" (UID: \"d7c0d558-7283-4304-a4a6-1a9a110ba265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24756d7a-pwrs7"
Oct 30 17:01:19.685461 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:19.685295    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kttzs\" (UniqueName: \"kubernetes.io/projected/d7c0d558-7283-4304-a4a6-1a9a110ba265-kube-api-access-kttzs\") pod \"maas-api-revoke-sub-24756d7a-pwrs7\" (UID: \"d7c0d558-7283-4304-a4a6-1a9a110ba265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24756d7a-pwrs7"
Oct 30 17:01:19.685536 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:19.685460    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7c0d558-7283-4304-a4a6-1a9a110ba265-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24756d7a-pwrs7\" (UID: \"d7c0d558-7283-4304-a4a6-1a9a110ba265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24756d7a-pwrs7"
Oct 30 17:01:19.686019 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:19.685993    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7c0d558-7283-4304-a4a6-1a9a110ba265-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24756d7a-pwrs7\" (UID: \"d7c0d558-7283-4304-a4a6-1a9a110ba265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24756d7a-pwrs7"
Oct 30 17:01:19.688844 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:19.688818    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7c0d558-7283-4304-a4a6-1a9a110ba265-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24756d7a-pwrs7\" (UID: \"d7c0d558-7283-4304-a4a6-1a9a110ba265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24756d7a-pwrs7"
Oct 30 17:01:19.709976 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:19.709953    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kttzs\" (UniqueName: \"kubernetes.io/projected/d7c0d558-7283-4304-a4a6-1a9a110ba265-kube-api-access-kttzs\") pod \"maas-api-revoke-sub-24756d7a-pwrs7\" (UID: \"d7c0d558-7283-4304-a4a6-1a9a110ba265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24756d7a-pwrs7"
Oct 30 17:01:19.747551 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:19.747463    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b089650-8x8fb"
Oct 30 17:01:19.747551 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:19.747462    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b089650-8x8fb" event={"ID":"08dd97a0-74fd-4b66-84ff-47717c5b8849","Type":"ContainerDied","Data":"6fc014244bb6b2f8e2eb5b9c77754edfc727896c045910a8c2137f331e568530"}
Oct 30 17:01:19.747761 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:19.747572    4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fc014244bb6b2f8e2eb5b9c77754edfc727896c045910a8c2137f331e568530"
Oct 30 17:01:19.748829 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:19.748804    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cfa206-5m555"
Oct 30 17:01:19.748923 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:19.748854    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cfa206-5m555" event={"ID":"8edded39-42a8-4d0b-bfb1-d5b77d3102fd","Type":"ContainerDied","Data":"87e30379b6eb0e82ed8216239d70527b648987aa4b8f7673ba71d1b75943ab6a"}
Oct 30 17:01:19.748923 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:19.748906    4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87e30379b6eb0e82ed8216239d70527b648987aa4b8f7673ba71d1b75943ab6a"
Oct 30 17:01:19.798607 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:19.798578    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24756d7a-pwrs7"
Oct 30 17:01:20.098918 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:20.098891    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24756d7a-pwrs7"]
Oct 30 17:01:20.101807 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:01:20.101779    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7c0d558_7283_4304_a4a6_1a9a110ba265.slice/crio-917d03d74ceba21ea232391949af580726a5c2e8e8a75c9bb4a846b7197c11ad WatchSource:0}: Error finding container 917d03d74ceba21ea232391949af580726a5c2e8e8a75c9bb4a846b7197c11ad: Status 404 returned error can't find the container with id 917d03d74ceba21ea232391949af580726a5c2e8e8a75c9bb4a846b7197c11ad
Oct 30 17:01:20.756713 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:20.756670    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24756d7a-pwrs7" event={"ID":"d7c0d558-7283-4304-a4a6-1a9a110ba265","Type":"ContainerStarted","Data":"f1ccb5bd653a9f08e9850696b0583acb0f6387e183f7e128f204b69e53ada26b"}
Oct 30 17:01:20.756921 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:20.756743    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24756d7a-pwrs7" event={"ID":"d7c0d558-7283-4304-a4a6-1a9a110ba265","Type":"ContainerStarted","Data":"917d03d74ceba21ea232391949af580726a5c2e8e8a75c9bb4a846b7197c11ad"}
Oct 30 17:01:20.802444 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:20.802359    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24756d7a-pwrs7" podStartSLOduration=1.802338839 podStartE2EDuration="1.802338839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:20.798578715 +0000 UTC m=+857.233480923" watchObservedRunningTime="2026-10-30 17:01:20.802338839 +0000 UTC m=+857.237241060"
Oct 30 17:01:21.769445 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:21.768625    4433 generic.go:356] "Generic (PLEG): container finished" podID="d7c0d558-7283-4304-a4a6-1a9a110ba265" containerID="f1ccb5bd653a9f08e9850696b0583acb0f6387e183f7e128f204b69e53ada26b" exitCode=0
Oct 30 17:01:21.769445 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:21.768790    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24756d7a-pwrs7" event={"ID":"d7c0d558-7283-4304-a4a6-1a9a110ba265","Type":"ContainerDied","Data":"f1ccb5bd653a9f08e9850696b0583acb0f6387e183f7e128f204b69e53ada26b"}
Oct 30 17:01:22.549878 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:22.549841    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"]
Oct 30 17:01:22.560751 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:22.560716    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"
Oct 30 17:01:22.580481 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:22.580447    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-86f50a-openshift-default-dockercfg-n29ph\""
Oct 30 17:01:22.597862 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:22.596919    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"]
Oct 30 17:01:22.628752 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:22.628715    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-istio-token\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-np479\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"
Oct 30 17:01:22.628752 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:22.628757    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-istiod-ca-cert\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-np479\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"
Oct 30 17:01:22.628969 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:22.628881    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp549\" (UniqueName: \"kubernetes.io/projected/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-kube-api-access-zp549\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-np479\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"
Oct 30 17:01:22.628969 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:22.628921    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-istio-data\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-np479\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"
Oct 30 17:01:22.628969 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:22.628952    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-credential-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-np479\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"
Oct 30 17:01:22.629074 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:22.629048    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-workload-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-np479\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"
Oct 30 17:01:22.629114 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:22.629072    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-istio-podinfo\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-np479\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"
Oct 30 17:01:22.629114 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:22.629106    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-istio-envoy\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-np479\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"
Oct 30 17:01:22.629187 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:22.629139    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-workload-certs\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-np479\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"
Oct 30 17:01:22.730337 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:22.730301    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp549\" (UniqueName: \"kubernetes.io/projected/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-kube-api-access-zp549\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-np479\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"
Oct 30 17:01:22.730337 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:22.730341    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-istio-data\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-np479\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"
Oct 30 17:01:22.730633 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:22.730358    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-credential-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-np479\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"
Oct 30 17:01:22.730633 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:22.730393    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-workload-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-np479\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"
Oct 30 17:01:22.730633 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:22.730454    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-istio-podinfo\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-np479\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"
Oct 30 17:01:22.730633 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:22.730509    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-istio-envoy\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-np479\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"
Oct 30 17:01:22.730633 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:22.730544    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-workload-certs\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-np479\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"
Oct 30 17:01:22.730891 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:22.730684    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-istio-token\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-np479\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"
Oct 30 17:01:22.730891 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:22.730717    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-istiod-ca-cert\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-np479\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"
Oct 30 17:01:22.730891 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:22.730802    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-workload-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-np479\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"
Oct 30 17:01:22.730891 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:22.730877    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-credential-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-np479\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"
Oct 30 17:01:22.731034 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:22.730929    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-istio-data\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-np479\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"
Oct 30 17:01:22.731034 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:22.730987    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-workload-certs\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-np479\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"
Oct 30 17:01:22.731334 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:22.731309    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-istiod-ca-cert\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-np479\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"
Oct 30 17:01:22.732900 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:22.732873    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-istio-envoy\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-np479\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"
Oct 30 17:01:22.733343 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:22.733321    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-istio-podinfo\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-np479\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"
Oct 30 17:01:22.755043 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:22.754967    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp549\" (UniqueName: \"kubernetes.io/projected/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-kube-api-access-zp549\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-np479\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"
Oct 30 17:01:22.780161 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:22.779364    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-istio-token\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-np479\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"
Oct 30 17:01:22.889958 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:22.889626    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"
Oct 30 17:01:23.009849 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:23.009816    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24756d7a-pwrs7"
Oct 30 17:01:23.136125 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:23.136019    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7c0d558-7283-4304-a4a6-1a9a110ba265-kube-api-access-kttzs\" (UniqueName: \"kubernetes.io/projected/d7c0d558-7283-4304-a4a6-1a9a110ba265-kube-api-access-kttzs\") pod \"d7c0d558-7283-4304-a4a6-1a9a110ba265\" (UID: \"d7c0d558-7283-4304-a4a6-1a9a110ba265\") "
Oct 30 17:01:23.136125 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:23.136083    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7c0d558-7283-4304-a4a6-1a9a110ba265-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7c0d558-7283-4304-a4a6-1a9a110ba265-maas-api-openshift-ca\") pod \"d7c0d558-7283-4304-a4a6-1a9a110ba265\" (UID: \"d7c0d558-7283-4304-a4a6-1a9a110ba265\") "
Oct 30 17:01:23.136358 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:23.136129    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7c0d558-7283-4304-a4a6-1a9a110ba265-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7c0d558-7283-4304-a4a6-1a9a110ba265-maas-api-xks-ca\") pod \"d7c0d558-7283-4304-a4a6-1a9a110ba265\" (UID: \"d7c0d558-7283-4304-a4a6-1a9a110ba265\") "
Oct 30 17:01:23.137134 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:23.136593    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7c0d558-7283-4304-a4a6-1a9a110ba265-maas-api-openshift-ca" pod "d7c0d558-7283-4304-a4a6-1a9a110ba265" (UID: "d7c0d558-7283-4304-a4a6-1a9a110ba265"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:01:23.137134 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:23.136789    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7c0d558-7283-4304-a4a6-1a9a110ba265-maas-api-openshift-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:01:23.138840 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:23.138809    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7c0d558-7283-4304-a4a6-1a9a110ba265-kube-api-access-kttzs" pod "d7c0d558-7283-4304-a4a6-1a9a110ba265" (UID: "d7c0d558-7283-4304-a4a6-1a9a110ba265"). InnerVolumeSpecName "kube-api-access-kttzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:01:23.139377 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:23.139350    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7c0d558-7283-4304-a4a6-1a9a110ba265-maas-api-xks-ca" pod "d7c0d558-7283-4304-a4a6-1a9a110ba265" (UID: "d7c0d558-7283-4304-a4a6-1a9a110ba265"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:01:23.238096 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:23.238062    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kttzs\" (UniqueName: \"kubernetes.io/projected/d7c0d558-7283-4304-a4a6-1a9a110ba265-kube-api-access-kttzs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:01:23.238096 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:23.238092    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7c0d558-7283-4304-a4a6-1a9a110ba265-maas-api-xks-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:01:23.291303 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:23.291253    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"]
Oct 30 17:01:23.294093 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:01:23.294067    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20dd24a1_9e9c_40d2_8ad8_bf0a206f292e.slice/crio-79a8f5d435ce1cae0a03b630f8d9a272322959768ac43cba897cb2a5d35c610e WatchSource:0}: Error finding container 79a8f5d435ce1cae0a03b630f8d9a272322959768ac43cba897cb2a5d35c610e: Status 404 returned error can't find the container with id 79a8f5d435ce1cae0a03b630f8d9a272322959768ac43cba897cb2a5d35c610e
Oct 30 17:01:23.296542 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:23.296511    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:01:23.296619 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:23.296579    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:01:23.296660 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:23.296630    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:01:23.784647 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:23.784615    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24756d7a-pwrs7"
Oct 30 17:01:23.784647 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:23.784636    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24756d7a-pwrs7" event={"ID":"d7c0d558-7283-4304-a4a6-1a9a110ba265","Type":"ContainerDied","Data":"917d03d74ceba21ea232391949af580726a5c2e8e8a75c9bb4a846b7197c11ad"}
Oct 30 17:01:23.785196 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:23.784694    4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="917d03d74ceba21ea232391949af580726a5c2e8e8a75c9bb4a846b7197c11ad"
Oct 30 17:01:23.786464 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:23.786379    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479" event={"ID":"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e","Type":"ContainerStarted","Data":"50f27108e1e0a0f9bf20bcfd12c55313d37647207fa3d6d2c4714cac324afed8"}
Oct 30 17:01:23.786464 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:23.786441    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479" event={"ID":"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e","Type":"ContainerStarted","Data":"79a8f5d435ce1cae0a03b630f8d9a272322959768ac43cba897cb2a5d35c610e"}
Oct 30 17:01:23.834121 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:23.834052    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479" podStartSLOduration=1.834031912 podStartE2EDuration="1.834031912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:23.830428209 +0000 UTC m=+860.265330417" watchObservedRunningTime="2026-10-30 17:01:23.834031912 +0000 UTC m=+860.268934120"
Oct 30 17:01:23.890201 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:23.890166    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"
Oct 30 17:01:24.892274 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:24.892233    4433 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 17:01:24.892687 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:24.892299    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479" podUID="20dd24a1-9e9c-40d2-8ad8-bf0a206f292e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 17:01:25.890949 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:25.890910    4433 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 17:01:25.891123 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:25.890968    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479" podUID="20dd24a1-9e9c-40d2-8ad8-bf0a206f292e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 17:01:26.890921 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:26.890883    4433 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 17:01:26.891458 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:26.890949    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479" podUID="20dd24a1-9e9c-40d2-8ad8-bf0a206f292e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 17:01:27.002906 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:01:27.002866    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:01:27.890527 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:27.890489    4433 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 17:01:27.890706 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:27.890546    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479" podUID="20dd24a1-9e9c-40d2-8ad8-bf0a206f292e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 17:01:28.890213 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:28.890176    4433 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 17:01:28.892752 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:28.890239    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479" podUID="20dd24a1-9e9c-40d2-8ad8-bf0a206f292e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 17:01:29.679065 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:29.679024    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-233c3d40-4prqw"]
Oct 30 17:01:29.679605 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:29.679585    4433 cpu_manager.go:414] "containerMap: removing container" podUID="d7c0d558-7283-4304-a4a6-1a9a110ba265" containerName="revoke-keys"
Oct 30 17:01:29.679705 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:29.679606    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7c0d558-7283-4304-a4a6-1a9a110ba265" containerName="revoke-keys"
Oct 30 17:01:29.679782 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:29.679769    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7c0d558-7283-4304-a4a6-1a9a110ba265" containerName="revoke-keys"
Oct 30 17:01:29.682954 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:29.682933    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-233c3d40-4prqw"
Oct 30 17:01:29.698395 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:29.698365    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:01:29.709499 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:29.709465    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\""
Oct 30 17:01:29.712146 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:29.712118    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-233c3d40-4prqw"]
Oct 30 17:01:29.834304 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:29.834264    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76f81542-3443-4db5-a900-f721f2e05e38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-233c3d40-4prqw\" (UID: \"76f81542-3443-4db5-a900-f721f2e05e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-233c3d40-4prqw"
Oct 30 17:01:29.834547 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:29.834324    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djm4b\" (UniqueName: \"kubernetes.io/projected/76f81542-3443-4db5-a900-f721f2e05e38-kube-api-access-djm4b\") pod \"maas-api-revoke-sub-233c3d40-4prqw\" (UID: \"76f81542-3443-4db5-a900-f721f2e05e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-233c3d40-4prqw"
Oct 30 17:01:29.834547 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:29.834526    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76f81542-3443-4db5-a900-f721f2e05e38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-233c3d40-4prqw\" (UID: \"76f81542-3443-4db5-a900-f721f2e05e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-233c3d40-4prqw"
Oct 30 17:01:29.936165 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:29.936072    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76f81542-3443-4db5-a900-f721f2e05e38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-233c3d40-4prqw\" (UID: \"76f81542-3443-4db5-a900-f721f2e05e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-233c3d40-4prqw"
Oct 30 17:01:29.936692 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:29.936205    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76f81542-3443-4db5-a900-f721f2e05e38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-233c3d40-4prqw\" (UID: \"76f81542-3443-4db5-a900-f721f2e05e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-233c3d40-4prqw"
Oct 30 17:01:29.936692 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:29.936249    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djm4b\" (UniqueName: \"kubernetes.io/projected/76f81542-3443-4db5-a900-f721f2e05e38-kube-api-access-djm4b\") pod \"maas-api-revoke-sub-233c3d40-4prqw\" (UID: \"76f81542-3443-4db5-a900-f721f2e05e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-233c3d40-4prqw"
Oct 30 17:01:29.937008 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:29.936987    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76f81542-3443-4db5-a900-f721f2e05e38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-233c3d40-4prqw\" (UID: \"76f81542-3443-4db5-a900-f721f2e05e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-233c3d40-4prqw"
Oct 30 17:01:29.939450 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:29.939421    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76f81542-3443-4db5-a900-f721f2e05e38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-233c3d40-4prqw\" (UID: \"76f81542-3443-4db5-a900-f721f2e05e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-233c3d40-4prqw"
Oct 30 17:01:29.959939 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:29.959911    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djm4b\" (UniqueName: \"kubernetes.io/projected/76f81542-3443-4db5-a900-f721f2e05e38-kube-api-access-djm4b\") pod \"maas-api-revoke-sub-233c3d40-4prqw\" (UID: \"76f81542-3443-4db5-a900-f721f2e05e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-233c3d40-4prqw"
Oct 30 17:01:30.002241 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:30.002202    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-233c3d40-4prqw"
Oct 30 17:01:30.149447 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:30.149006    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"
Oct 30 17:01:30.149896 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:30.149866    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"
Oct 30 17:01:30.154455 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:30.154421    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"
Oct 30 17:01:30.347706 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:30.347624    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-233c3d40-4prqw"]
Oct 30 17:01:30.356003 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:01:30.355962    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76f81542_3443_4db5_a900_f721f2e05e38.slice/crio-836ad86da22b810155752febb90773de4050fdba8d45a6a72f97810827787ab0 WatchSource:0}: Error finding container 836ad86da22b810155752febb90773de4050fdba8d45a6a72f97810827787ab0: Status 404 returned error can't find the container with id 836ad86da22b810155752febb90773de4050fdba8d45a6a72f97810827787ab0
Oct 30 17:01:30.825933 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:30.825891    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-233c3d40-4prqw" event={"ID":"76f81542-3443-4db5-a900-f721f2e05e38","Type":"ContainerStarted","Data":"b41023255b0261bbc786418db05a71ff1fe0e04b0843663de3f5abceb958c6a2"}
Oct 30 17:01:30.826126 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:30.825957    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-233c3d40-4prqw" event={"ID":"76f81542-3443-4db5-a900-f721f2e05e38","Type":"ContainerStarted","Data":"836ad86da22b810155752febb90773de4050fdba8d45a6a72f97810827787ab0"}
Oct 30 17:01:30.867076 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:30.867010    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-233c3d40-4prqw" podStartSLOduration=1.866990597 podStartE2EDuration="1.866990597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:30.865502054 +0000 UTC m=+867.300404262" watchObservedRunningTime="2026-10-30 17:01:30.866990597 +0000 UTC m=+867.301892805"
Oct 30 17:01:31.484814 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:31.484775    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"]
Oct 30 17:01:31.488811 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:31.488781    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:01:31.529879 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:31.518236    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-d384dc\"/\"kube-root-ca.crt\""
Oct 30 17:01:31.529879 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:31.518444    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-d384dc\"/\"openshift-service-ca.crt\""
Oct 30 17:01:31.529879 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:31.518464    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-d384dc\"/\"distinct-w6-d384dc-kserve-self-signed-certs\""
Oct 30 17:01:31.529879 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:31.518533    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-d384dc\"/\"default-dockercfg-vqwcf\""
Oct 30 17:01:31.529879 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:31.522577    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"]
Oct 30 17:01:31.554347 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:31.554315    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb78a5ca-3157-4707-a75f-f38748185b81-home\") pod \"distinct-w6-d384dc-kserve-88966dd47-6skpg\" (UID: \"bb78a5ca-3157-4707-a75f-f38748185b81\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:01:31.554527 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:31.554353    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb78a5ca-3157-4707-a75f-f38748185b81-model-cache\") pod \"distinct-w6-d384dc-kserve-88966dd47-6skpg\" (UID: \"bb78a5ca-3157-4707-a75f-f38748185b81\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:01:31.554527 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:31.554478    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb78a5ca-3157-4707-a75f-f38748185b81-dshm\") pod \"distinct-w6-d384dc-kserve-88966dd47-6skpg\" (UID: \"bb78a5ca-3157-4707-a75f-f38748185b81\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:01:31.554649 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:31.554580    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb78a5ca-3157-4707-a75f-f38748185b81-tls-certs\") pod \"distinct-w6-d384dc-kserve-88966dd47-6skpg\" (UID: \"bb78a5ca-3157-4707-a75f-f38748185b81\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:01:31.554649 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:31.554630    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g86z\" (UniqueName: \"kubernetes.io/projected/bb78a5ca-3157-4707-a75f-f38748185b81-kube-api-access-7g86z\") pod \"distinct-w6-d384dc-kserve-88966dd47-6skpg\" (UID: \"bb78a5ca-3157-4707-a75f-f38748185b81\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:01:31.554732 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:31.554651    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb78a5ca-3157-4707-a75f-f38748185b81-tmp-dir\") pod \"distinct-w6-d384dc-kserve-88966dd47-6skpg\" (UID: \"bb78a5ca-3157-4707-a75f-f38748185b81\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:01:31.656381 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:31.656326    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb78a5ca-3157-4707-a75f-f38748185b81-home\") pod \"distinct-w6-d384dc-kserve-88966dd47-6skpg\" (UID: \"bb78a5ca-3157-4707-a75f-f38748185b81\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:01:31.656605 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:31.656395    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb78a5ca-3157-4707-a75f-f38748185b81-model-cache\") pod \"distinct-w6-d384dc-kserve-88966dd47-6skpg\" (UID: \"bb78a5ca-3157-4707-a75f-f38748185b81\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:01:31.656605 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:31.656521    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb78a5ca-3157-4707-a75f-f38748185b81-dshm\") pod \"distinct-w6-d384dc-kserve-88966dd47-6skpg\" (UID: \"bb78a5ca-3157-4707-a75f-f38748185b81\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:01:31.656605 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:31.656563    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb78a5ca-3157-4707-a75f-f38748185b81-tls-certs\") pod \"distinct-w6-d384dc-kserve-88966dd47-6skpg\" (UID: \"bb78a5ca-3157-4707-a75f-f38748185b81\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:01:31.656605 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:31.656592    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g86z\" (UniqueName: \"kubernetes.io/projected/bb78a5ca-3157-4707-a75f-f38748185b81-kube-api-access-7g86z\") pod \"distinct-w6-d384dc-kserve-88966dd47-6skpg\" (UID: \"bb78a5ca-3157-4707-a75f-f38748185b81\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:01:31.656828 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:31.656621    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb78a5ca-3157-4707-a75f-f38748185b81-tmp-dir\") pod \"distinct-w6-d384dc-kserve-88966dd47-6skpg\" (UID: \"bb78a5ca-3157-4707-a75f-f38748185b81\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:01:31.656828 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:31.656794    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb78a5ca-3157-4707-a75f-f38748185b81-home\") pod \"distinct-w6-d384dc-kserve-88966dd47-6skpg\" (UID: \"bb78a5ca-3157-4707-a75f-f38748185b81\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:01:31.657139 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:31.657111    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb78a5ca-3157-4707-a75f-f38748185b81-model-cache\") pod \"distinct-w6-d384dc-kserve-88966dd47-6skpg\" (UID: \"bb78a5ca-3157-4707-a75f-f38748185b81\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:01:31.657266 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:31.657245    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb78a5ca-3157-4707-a75f-f38748185b81-tmp-dir\") pod \"distinct-w6-d384dc-kserve-88966dd47-6skpg\" (UID: \"bb78a5ca-3157-4707-a75f-f38748185b81\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:01:31.658818 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:31.658793    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb78a5ca-3157-4707-a75f-f38748185b81-dshm\") pod \"distinct-w6-d384dc-kserve-88966dd47-6skpg\" (UID: \"bb78a5ca-3157-4707-a75f-f38748185b81\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:01:31.659390 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:31.659369    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb78a5ca-3157-4707-a75f-f38748185b81-tls-certs\") pod \"distinct-w6-d384dc-kserve-88966dd47-6skpg\" (UID: \"bb78a5ca-3157-4707-a75f-f38748185b81\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:01:31.682353 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:31.682318    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g86z\" (UniqueName: \"kubernetes.io/projected/bb78a5ca-3157-4707-a75f-f38748185b81-kube-api-access-7g86z\") pod \"distinct-w6-d384dc-kserve-88966dd47-6skpg\" (UID: \"bb78a5ca-3157-4707-a75f-f38748185b81\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:01:31.804485 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:31.804370    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:01:31.833173 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:31.833139    4433 generic.go:356] "Generic (PLEG): container finished" podID="76f81542-3443-4db5-a900-f721f2e05e38" containerID="b41023255b0261bbc786418db05a71ff1fe0e04b0843663de3f5abceb958c6a2" exitCode=0
Oct 30 17:01:31.833319 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:31.833221    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-233c3d40-4prqw" event={"ID":"76f81542-3443-4db5-a900-f721f2e05e38","Type":"ContainerDied","Data":"b41023255b0261bbc786418db05a71ff1fe0e04b0843663de3f5abceb958c6a2"}
Oct 30 17:01:32.109395 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:32.109368    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"]
Oct 30 17:01:32.111158 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:01:32.111119    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb78a5ca_3157_4707_a75f_f38748185b81.slice/crio-4ea30057cf180ff68e72643094aef8b1125c3a21a98bf9b4df51f257e5e1a131 WatchSource:0}: Error finding container 4ea30057cf180ff68e72643094aef8b1125c3a21a98bf9b4df51f257e5e1a131: Status 404 returned error can't find the container with id 4ea30057cf180ff68e72643094aef8b1125c3a21a98bf9b4df51f257e5e1a131
Oct 30 17:01:32.840645 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:32.840591    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" event={"ID":"bb78a5ca-3157-4707-a75f-f38748185b81","Type":"ContainerStarted","Data":"a2dea290264a109e51f06f3e1e92c061ed6a1e4619e214b4b9269d75ff04e062"}
Oct 30 17:01:32.841140 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:32.840660    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" event={"ID":"bb78a5ca-3157-4707-a75f-f38748185b81","Type":"ContainerStarted","Data":"4ea30057cf180ff68e72643094aef8b1125c3a21a98bf9b4df51f257e5e1a131"}
Oct 30 17:01:32.887462 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:32.887383    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podStartSLOduration=1.887362728 podStartE2EDuration="1.887362728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:32.883997381 +0000 UTC m=+869.318899588" watchObservedRunningTime="2026-10-30 17:01:32.887362728 +0000 UTC m=+869.322264935"
Oct 30 17:01:32.990309 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:32.990279    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-233c3d40-4prqw"
Oct 30 17:01:33.074658 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:33.074624    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76f81542-3443-4db5-a900-f721f2e05e38-kube-api-access-djm4b\" (UniqueName: \"kubernetes.io/projected/76f81542-3443-4db5-a900-f721f2e05e38-kube-api-access-djm4b\") pod \"76f81542-3443-4db5-a900-f721f2e05e38\" (UID: \"76f81542-3443-4db5-a900-f721f2e05e38\") "
Oct 30 17:01:33.074881 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:33.074706    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76f81542-3443-4db5-a900-f721f2e05e38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76f81542-3443-4db5-a900-f721f2e05e38-maas-api-openshift-ca\") pod \"76f81542-3443-4db5-a900-f721f2e05e38\" (UID: \"76f81542-3443-4db5-a900-f721f2e05e38\") "
Oct 30 17:01:33.074881 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:33.074789    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76f81542-3443-4db5-a900-f721f2e05e38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76f81542-3443-4db5-a900-f721f2e05e38-maas-api-xks-ca\") pod \"76f81542-3443-4db5-a900-f721f2e05e38\" (UID: \"76f81542-3443-4db5-a900-f721f2e05e38\") "
Oct 30 17:01:33.075079 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:33.075053    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76f81542-3443-4db5-a900-f721f2e05e38-maas-api-openshift-ca" pod "76f81542-3443-4db5-a900-f721f2e05e38" (UID: "76f81542-3443-4db5-a900-f721f2e05e38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:01:33.075286 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:33.075265    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76f81542-3443-4db5-a900-f721f2e05e38-maas-api-openshift-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:01:33.076945 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:33.076913    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76f81542-3443-4db5-a900-f721f2e05e38-kube-api-access-djm4b" pod "76f81542-3443-4db5-a900-f721f2e05e38" (UID: "76f81542-3443-4db5-a900-f721f2e05e38"). InnerVolumeSpecName "kube-api-access-djm4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:01:33.077118 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:33.077101    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76f81542-3443-4db5-a900-f721f2e05e38-maas-api-xks-ca" pod "76f81542-3443-4db5-a900-f721f2e05e38" (UID: "76f81542-3443-4db5-a900-f721f2e05e38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:01:33.176445 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:33.176370    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djm4b\" (UniqueName: \"kubernetes.io/projected/76f81542-3443-4db5-a900-f721f2e05e38-kube-api-access-djm4b\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:01:33.176445 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:33.176417    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76f81542-3443-4db5-a900-f721f2e05e38-maas-api-xks-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:01:33.804803 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:33.804769    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:01:33.847175 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:33.847141    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-233c3d40-4prqw"
Oct 30 17:01:33.847175 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:33.847169    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-233c3d40-4prqw" event={"ID":"76f81542-3443-4db5-a900-f721f2e05e38","Type":"ContainerDied","Data":"836ad86da22b810155752febb90773de4050fdba8d45a6a72f97810827787ab0"}
Oct 30 17:01:33.847672 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:33.847218    4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="836ad86da22b810155752febb90773de4050fdba8d45a6a72f97810827787ab0"
Oct 30 17:01:35.920540 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:35.920510    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:01:37.870699 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:37.870665    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"]
Oct 30 17:01:37.872044 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:37.872016    4433 cpu_manager.go:414] "containerMap: removing container" podUID="76f81542-3443-4db5-a900-f721f2e05e38" containerName="revoke-keys"
Oct 30 17:01:37.872219 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:37.872203    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76f81542-3443-4db5-a900-f721f2e05e38" containerName="revoke-keys"
Oct 30 17:01:37.872467 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:37.872450    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="76f81542-3443-4db5-a900-f721f2e05e38" containerName="revoke-keys"
Oct 30 17:01:37.885430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:37.883060    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"
Oct 30 17:01:37.897764 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:37.897702    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"]
Oct 30 17:01:37.899791 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:37.899753    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-096d75-openshift-default-dockercfg-twcrb\""
Oct 30 17:01:38.034646 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:38.034606    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41ff63dc-b547-4143-adb5-d904bc6e91d8-istio-data\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"
Oct 30 17:01:38.034850 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:38.034749    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41ff63dc-b547-4143-adb5-d904bc6e91d8-istiod-ca-cert\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"
Oct 30 17:01:38.034850 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:38.034819    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ff63dc-b547-4143-adb5-d904bc6e91d8-workload-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"
Oct 30 17:01:38.034977 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:38.034894    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrprp\" (UniqueName: \"kubernetes.io/projected/41ff63dc-b547-4143-adb5-d904bc6e91d8-kube-api-access-nrprp\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"
Oct 30 17:01:38.034977 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:38.034951    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41ff63dc-b547-4143-adb5-d904bc6e91d8-istio-podinfo\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"
Oct 30 17:01:38.035085 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:38.035030    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ff63dc-b547-4143-adb5-d904bc6e91d8-credential-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"
Oct 30 17:01:38.035085 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:38.035055    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41ff63dc-b547-4143-adb5-d904bc6e91d8-workload-certs\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"
Oct 30 17:01:38.035192 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:38.035129    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41ff63dc-b547-4143-adb5-d904bc6e91d8-istio-token\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"
Oct 30 17:01:38.035192 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:38.035161    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41ff63dc-b547-4143-adb5-d904bc6e91d8-istio-envoy\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"
Oct 30 17:01:38.136758 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:38.136650    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41ff63dc-b547-4143-adb5-d904bc6e91d8-istio-token\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"
Oct 30 17:01:38.136758 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:38.136710    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41ff63dc-b547-4143-adb5-d904bc6e91d8-istio-envoy\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"
Oct 30 17:01:38.136758 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:38.136749    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41ff63dc-b547-4143-adb5-d904bc6e91d8-istio-data\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"
Oct 30 17:01:38.137048 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:38.136882    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41ff63dc-b547-4143-adb5-d904bc6e91d8-istiod-ca-cert\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"
Oct 30 17:01:38.137048 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:38.136975    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ff63dc-b547-4143-adb5-d904bc6e91d8-workload-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"
Oct 30 17:01:38.137150 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:38.137073    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrprp\" (UniqueName: \"kubernetes.io/projected/41ff63dc-b547-4143-adb5-d904bc6e91d8-kube-api-access-nrprp\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"
Oct 30 17:01:38.137150 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:38.137136    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41ff63dc-b547-4143-adb5-d904bc6e91d8-istio-data\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"
Oct 30 17:01:38.137253 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:38.137142    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41ff63dc-b547-4143-adb5-d904bc6e91d8-istio-podinfo\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"
Oct 30 17:01:38.137519 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:38.137489    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ff63dc-b547-4143-adb5-d904bc6e91d8-credential-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"
Oct 30 17:01:38.137644 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:38.137530    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41ff63dc-b547-4143-adb5-d904bc6e91d8-workload-certs\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"
Oct 30 17:01:38.138187 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:38.138130    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ff63dc-b547-4143-adb5-d904bc6e91d8-credential-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"
Oct 30 17:01:38.138187 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:38.138138    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41ff63dc-b547-4143-adb5-d904bc6e91d8-workload-certs\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"
Oct 30 17:01:38.138350 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:38.138194    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ff63dc-b547-4143-adb5-d904bc6e91d8-workload-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"
Oct 30 17:01:38.138606 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:38.138585    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41ff63dc-b547-4143-adb5-d904bc6e91d8-istiod-ca-cert\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"
Oct 30 17:01:38.145161 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:38.145124    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41ff63dc-b547-4143-adb5-d904bc6e91d8-istio-envoy\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"
Oct 30 17:01:38.145332 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:38.145308    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41ff63dc-b547-4143-adb5-d904bc6e91d8-istio-podinfo\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"
Oct 30 17:01:38.190430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:38.187622    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41ff63dc-b547-4143-adb5-d904bc6e91d8-istio-token\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"
Oct 30 17:01:38.230427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:38.230205    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrprp\" (UniqueName: \"kubernetes.io/projected/41ff63dc-b547-4143-adb5-d904bc6e91d8-kube-api-access-nrprp\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"
Oct 30 17:01:38.236438 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:38.235566    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"
Oct 30 17:01:38.824557 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:38.824367    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"]
Oct 30 17:01:38.826054 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:01:38.826016    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41ff63dc_b547_4143_adb5_d904bc6e91d8.slice/crio-68c7814d00e519dfbfe06fa058e0a588ebd9c5ecfa48dd5302bcf01375f71932 WatchSource:0}: Error finding container 68c7814d00e519dfbfe06fa058e0a588ebd9c5ecfa48dd5302bcf01375f71932: Status 404 returned error can't find the container with id 68c7814d00e519dfbfe06fa058e0a588ebd9c5ecfa48dd5302bcf01375f71932
Oct 30 17:01:38.831437 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:38.831169    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:01:38.831437 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:38.831272    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:01:38.831437 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:38.831321    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:01:38.890369 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:38.890333    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9" event={"ID":"41ff63dc-b547-4143-adb5-d904bc6e91d8","Type":"ContainerStarted","Data":"68c7814d00e519dfbfe06fa058e0a588ebd9c5ecfa48dd5302bcf01375f71932"}
Oct 30 17:01:39.898539 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:39.898499    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9" event={"ID":"41ff63dc-b547-4143-adb5-d904bc6e91d8","Type":"ContainerStarted","Data":"9793b76780896f189144a1b76124914e6bffebd698d34db7e905ed9439eff0b2"}
Oct 30 17:01:39.986059 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:39.985988    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9" podStartSLOduration=2.985968313 podStartE2EDuration="2.985968313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:39.981970516 +0000 UTC m=+876.416872723" watchObservedRunningTime="2026-10-30 17:01:39.985968313 +0000 UTC m=+876.420870520"
Oct 30 17:01:40.236263 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:40.236222    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"
Oct 30 17:01:40.237909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:40.237883    4433 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 17:01:40.238048 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:40.237938    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9" podUID="41ff63dc-b547-4143-adb5-d904bc6e91d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 17:01:41.236735 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:41.236700    4433 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 17:01:41.237180 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:41.236759    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9" podUID="41ff63dc-b547-4143-adb5-d904bc6e91d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 17:01:41.805130 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:41.805096    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:01:41.817910 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:41.817880    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:01:41.920025 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:41.919993    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:01:42.237107 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:42.237063    4433 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 17:01:42.237525 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:42.237134    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9" podUID="41ff63dc-b547-4143-adb5-d904bc6e91d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 17:01:43.236753 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:43.236718    4433 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 17:01:43.236925 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:43.236815    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9" podUID="41ff63dc-b547-4143-adb5-d904bc6e91d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 17:01:44.236440 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:44.236375    4433 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 17:01:44.236843 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:44.236459    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9" podUID="41ff63dc-b547-4143-adb5-d904bc6e91d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 17:01:45.236509 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:45.236470    4433 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 17:01:45.236923 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:45.236539    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9" podUID="41ff63dc-b547-4143-adb5-d904bc6e91d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 17:01:46.241855 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:46.241824    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"
Oct 30 17:01:46.242334 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:46.242311    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"
Oct 30 17:01:46.242998 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:46.242978    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"
Oct 30 17:01:48.859789 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:48.859747    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"]
Oct 30 17:01:48.865024 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:48.864994    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"
Oct 30 17:01:48.924074 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:48.924038    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"]
Oct 30 17:01:48.964761 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:48.964723    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c986e3ae-ffab-420f-92ca-8234d2d918f6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n9gd9\" (UID: \"c986e3ae-ffab-420f-92ca-8234d2d918f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"
Oct 30 17:01:48.964928 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:48.964786    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c986e3ae-ffab-420f-92ca-8234d2d918f6-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n9gd9\" (UID: \"c986e3ae-ffab-420f-92ca-8234d2d918f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"
Oct 30 17:01:48.964928 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:48.964865    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c986e3ae-ffab-420f-92ca-8234d2d918f6-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n9gd9\" (UID: \"c986e3ae-ffab-420f-92ca-8234d2d918f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"
Oct 30 17:01:48.964928 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:48.964912    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w86xm\" (UniqueName: \"kubernetes.io/projected/c986e3ae-ffab-420f-92ca-8234d2d918f6-kube-api-access-w86xm\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n9gd9\" (UID: \"c986e3ae-ffab-420f-92ca-8234d2d918f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"
Oct 30 17:01:48.965093 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:48.965025    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c986e3ae-ffab-420f-92ca-8234d2d918f6-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n9gd9\" (UID: \"c986e3ae-ffab-420f-92ca-8234d2d918f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"
Oct 30 17:01:48.965093 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:48.965076    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c986e3ae-ffab-420f-92ca-8234d2d918f6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n9gd9\" (UID: \"c986e3ae-ffab-420f-92ca-8234d2d918f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"
Oct 30 17:01:48.965168 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:48.965130    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c986e3ae-ffab-420f-92ca-8234d2d918f6-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n9gd9\" (UID: \"c986e3ae-ffab-420f-92ca-8234d2d918f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"
Oct 30 17:01:48.965219 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:48.965202    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c986e3ae-ffab-420f-92ca-8234d2d918f6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n9gd9\" (UID: \"c986e3ae-ffab-420f-92ca-8234d2d918f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"
Oct 30 17:01:48.965311 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:48.965296    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c986e3ae-ffab-420f-92ca-8234d2d918f6-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n9gd9\" (UID: \"c986e3ae-ffab-420f-92ca-8234d2d918f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"
Oct 30 17:01:49.066745 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:49.066713    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c986e3ae-ffab-420f-92ca-8234d2d918f6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n9gd9\" (UID: \"c986e3ae-ffab-420f-92ca-8234d2d918f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"
Oct 30 17:01:49.066946 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:49.066764    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c986e3ae-ffab-420f-92ca-8234d2d918f6-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n9gd9\" (UID: \"c986e3ae-ffab-420f-92ca-8234d2d918f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"
Oct 30 17:01:49.066946 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:49.066780    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c986e3ae-ffab-420f-92ca-8234d2d918f6-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n9gd9\" (UID: \"c986e3ae-ffab-420f-92ca-8234d2d918f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"
Oct 30 17:01:49.066946 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:49.066810    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w86xm\" (UniqueName: \"kubernetes.io/projected/c986e3ae-ffab-420f-92ca-8234d2d918f6-kube-api-access-w86xm\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n9gd9\" (UID: \"c986e3ae-ffab-420f-92ca-8234d2d918f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"
Oct 30 17:01:49.066946 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:49.066848    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c986e3ae-ffab-420f-92ca-8234d2d918f6-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n9gd9\" (UID: \"c986e3ae-ffab-420f-92ca-8234d2d918f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"
Oct 30 17:01:49.066946 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:49.066874    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c986e3ae-ffab-420f-92ca-8234d2d918f6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n9gd9\" (UID: \"c986e3ae-ffab-420f-92ca-8234d2d918f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"
Oct 30 17:01:49.066946 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:49.066894    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c986e3ae-ffab-420f-92ca-8234d2d918f6-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n9gd9\" (UID: \"c986e3ae-ffab-420f-92ca-8234d2d918f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"
Oct 30 17:01:49.067188 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:49.067008    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c986e3ae-ffab-420f-92ca-8234d2d918f6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n9gd9\" (UID: \"c986e3ae-ffab-420f-92ca-8234d2d918f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"
Oct 30 17:01:49.067188 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:49.067054    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c986e3ae-ffab-420f-92ca-8234d2d918f6-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n9gd9\" (UID: \"c986e3ae-ffab-420f-92ca-8234d2d918f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"
Oct 30 17:01:49.067264 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:49.067194    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c986e3ae-ffab-420f-92ca-8234d2d918f6-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n9gd9\" (UID: \"c986e3ae-ffab-420f-92ca-8234d2d918f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"
Oct 30 17:01:49.067264 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:49.067211    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c986e3ae-ffab-420f-92ca-8234d2d918f6-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n9gd9\" (UID: \"c986e3ae-ffab-420f-92ca-8234d2d918f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"
Oct 30 17:01:49.067740 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:49.067716    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c986e3ae-ffab-420f-92ca-8234d2d918f6-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n9gd9\" (UID: \"c986e3ae-ffab-420f-92ca-8234d2d918f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"
Oct 30 17:01:49.067850 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:49.067752    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c986e3ae-ffab-420f-92ca-8234d2d918f6-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n9gd9\" (UID: \"c986e3ae-ffab-420f-92ca-8234d2d918f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"
Oct 30 17:01:49.067901 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:49.067850    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c986e3ae-ffab-420f-92ca-8234d2d918f6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n9gd9\" (UID: \"c986e3ae-ffab-420f-92ca-8234d2d918f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"
Oct 30 17:01:49.069504 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:49.069475    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c986e3ae-ffab-420f-92ca-8234d2d918f6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n9gd9\" (UID: \"c986e3ae-ffab-420f-92ca-8234d2d918f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"
Oct 30 17:01:49.069602 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:49.069529    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c986e3ae-ffab-420f-92ca-8234d2d918f6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n9gd9\" (UID: \"c986e3ae-ffab-420f-92ca-8234d2d918f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"
Oct 30 17:01:49.089882 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:49.089847    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c986e3ae-ffab-420f-92ca-8234d2d918f6-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n9gd9\" (UID: \"c986e3ae-ffab-420f-92ca-8234d2d918f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"
Oct 30 17:01:49.090030 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:49.090009    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w86xm\" (UniqueName: \"kubernetes.io/projected/c986e3ae-ffab-420f-92ca-8234d2d918f6-kube-api-access-w86xm\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n9gd9\" (UID: \"c986e3ae-ffab-420f-92ca-8234d2d918f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"
Oct 30 17:01:49.181207 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:49.181174    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"
Oct 30 17:01:49.502144 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:49.502113    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"]
Oct 30 17:01:49.502245 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:01:49.502171    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc986e3ae_ffab_420f_92ca_8234d2d918f6.slice/crio-d86a561561716b8b6c0a2a721bf4c77d62f2e687954c3f578d59492fb656ff99 WatchSource:0}: Error finding container d86a561561716b8b6c0a2a721bf4c77d62f2e687954c3f578d59492fb656ff99: Status 404 returned error can't find the container with id d86a561561716b8b6c0a2a721bf4c77d62f2e687954c3f578d59492fb656ff99
Oct 30 17:01:49.504526 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:49.504493    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:01:49.504619 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:49.504557    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:01:49.504619 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:49.504592    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:01:49.953345 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:49.953311    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9" event={"ID":"c986e3ae-ffab-420f-92ca-8234d2d918f6","Type":"ContainerStarted","Data":"c8a3202ab611b8e4093057d3cca9cabd078f6bb790089a9220ae55f862428bd8"}
Oct 30 17:01:49.953792 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:49.953364    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9" event={"ID":"c986e3ae-ffab-420f-92ca-8234d2d918f6","Type":"ContainerStarted","Data":"d86a561561716b8b6c0a2a721bf4c77d62f2e687954c3f578d59492fb656ff99"}
Oct 30 17:01:50.000485 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:50.000378    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9" podStartSLOduration=2.000361339 podStartE2EDuration="2.000361339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:49.998114595 +0000 UTC m=+886.433016805" watchObservedRunningTime="2026-10-30 17:01:50.000361339 +0000 UTC m=+886.435263618"
Oct 30 17:01:50.181962 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:50.181929    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"
Oct 30 17:01:50.184071 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:50.184037    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n9gd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 17:01:50.184231 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:50.184092    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9" podUID="c986e3ae-ffab-420f-92ca-8234d2d918f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 17:01:51.182007 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:51.181962    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n9gd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 17:01:51.182422 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:51.182024    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9" podUID="c986e3ae-ffab-420f-92ca-8234d2d918f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 17:01:52.007589 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.007552    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"]
Oct 30 17:01:52.011752 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.011723    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"
Oct 30 17:01:52.040553 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.040518    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b80d83-openshift-default-dockercfg-6zl9l\""
Oct 30 17:01:52.045370 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.045337    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"]
Oct 30 17:01:52.118179 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.118135    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7cd3d7-ced4-415b-b6c8-eb31a545daac-credential-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"
Oct 30 17:01:52.119373 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.119344    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7cd3d7-ced4-415b-b6c8-eb31a545daac-workload-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"
Oct 30 17:01:52.119750 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.119728    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df7cd3d7-ced4-415b-b6c8-eb31a545daac-istio-podinfo\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"
Oct 30 17:01:52.119920 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.119904    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df7cd3d7-ced4-415b-b6c8-eb31a545daac-istio-data\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"
Oct 30 17:01:52.120030 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.120015    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df7cd3d7-ced4-415b-b6c8-eb31a545daac-istio-token\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"
Oct 30 17:01:52.123501 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.123475    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df7cd3d7-ced4-415b-b6c8-eb31a545daac-istio-envoy\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"
Oct 30 17:01:52.123684 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.123665    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk87d\" (UniqueName: \"kubernetes.io/projected/df7cd3d7-ced4-415b-b6c8-eb31a545daac-kube-api-access-bk87d\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"
Oct 30 17:01:52.133572 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.133535    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df7cd3d7-ced4-415b-b6c8-eb31a545daac-workload-certs\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"
Oct 30 17:01:52.141487 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.137922    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df7cd3d7-ced4-415b-b6c8-eb31a545daac-istiod-ca-cert\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"
Oct 30 17:01:52.190832 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.190787    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n9gd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 17:01:52.191345 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.190856    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9" podUID="c986e3ae-ffab-420f-92ca-8234d2d918f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 17:01:52.242964 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.242932    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7cd3d7-ced4-415b-b6c8-eb31a545daac-credential-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"
Oct 30 17:01:52.242964 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.242968    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7cd3d7-ced4-415b-b6c8-eb31a545daac-workload-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"
Oct 30 17:01:52.243175 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.243036    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df7cd3d7-ced4-415b-b6c8-eb31a545daac-istio-podinfo\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"
Oct 30 17:01:52.243175 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.243056    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df7cd3d7-ced4-415b-b6c8-eb31a545daac-istio-data\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"
Oct 30 17:01:52.243175 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.243072    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df7cd3d7-ced4-415b-b6c8-eb31a545daac-istio-token\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"
Oct 30 17:01:52.243175 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.243104    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df7cd3d7-ced4-415b-b6c8-eb31a545daac-istio-envoy\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"
Oct 30 17:01:52.243175 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.243150    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk87d\" (UniqueName: \"kubernetes.io/projected/df7cd3d7-ced4-415b-b6c8-eb31a545daac-kube-api-access-bk87d\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"
Oct 30 17:01:52.243333 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.243197    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df7cd3d7-ced4-415b-b6c8-eb31a545daac-workload-certs\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"
Oct 30 17:01:52.243333 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.243246    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df7cd3d7-ced4-415b-b6c8-eb31a545daac-istiod-ca-cert\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"
Oct 30 17:01:52.243463 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.243438    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7cd3d7-ced4-415b-b6c8-eb31a545daac-credential-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"
Oct 30 17:01:52.243529 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.243494    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7cd3d7-ced4-415b-b6c8-eb31a545daac-workload-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"
Oct 30 17:01:52.243588 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.243546    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df7cd3d7-ced4-415b-b6c8-eb31a545daac-istio-data\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"
Oct 30 17:01:52.243769 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.243742    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df7cd3d7-ced4-415b-b6c8-eb31a545daac-workload-certs\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"
Oct 30 17:01:52.244063 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.244042    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df7cd3d7-ced4-415b-b6c8-eb31a545daac-istiod-ca-cert\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"
Oct 30 17:01:52.245531 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.245505    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df7cd3d7-ced4-415b-b6c8-eb31a545daac-istio-envoy\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"
Oct 30 17:01:52.245805 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.245789    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df7cd3d7-ced4-415b-b6c8-eb31a545daac-istio-podinfo\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"
Oct 30 17:01:52.276132 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.276056    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df7cd3d7-ced4-415b-b6c8-eb31a545daac-istio-token\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"
Oct 30 17:01:52.282321 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.282252    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk87d\" (UniqueName: \"kubernetes.io/projected/df7cd3d7-ced4-415b-b6c8-eb31a545daac-kube-api-access-bk87d\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"
Oct 30 17:01:52.308846 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.308814    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"]
Oct 30 17:01:52.313860 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.313838    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"
Oct 30 17:01:52.328340 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.328314    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"
Oct 30 17:01:52.343341 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.343310    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"]
Oct 30 17:01:52.446357 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.445838    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b04a22a-0651-49fe-9d44-7c0067c2ad14-istio-data\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"
Oct 30 17:01:52.446357 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.445930    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b04a22a-0651-49fe-9d44-7c0067c2ad14-istio-podinfo\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"
Oct 30 17:01:52.446357 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.445993    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b04a22a-0651-49fe-9d44-7c0067c2ad14-workload-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"
Oct 30 17:01:52.446357 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.446049    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b04a22a-0651-49fe-9d44-7c0067c2ad14-credential-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"
Oct 30 17:01:52.446357 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.446108    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b04a22a-0651-49fe-9d44-7c0067c2ad14-istiod-ca-cert\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"
Oct 30 17:01:52.446357 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.446142    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b04a22a-0651-49fe-9d44-7c0067c2ad14-workload-certs\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"
Oct 30 17:01:52.446357 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.446187    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b04a22a-0651-49fe-9d44-7c0067c2ad14-istio-envoy\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"
Oct 30 17:01:52.446357 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.446228    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nknwp\" (UniqueName: \"kubernetes.io/projected/5b04a22a-0651-49fe-9d44-7c0067c2ad14-kube-api-access-nknwp\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"
Oct 30 17:01:52.446803 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.446498    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b04a22a-0651-49fe-9d44-7c0067c2ad14-istio-token\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"
Oct 30 17:01:52.548166 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.548074    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b04a22a-0651-49fe-9d44-7c0067c2ad14-istio-podinfo\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"
Oct 30 17:01:52.548166 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.548141    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b04a22a-0651-49fe-9d44-7c0067c2ad14-workload-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"
Oct 30 17:01:52.548423 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.548188    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b04a22a-0651-49fe-9d44-7c0067c2ad14-credential-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"
Oct 30 17:01:52.548423 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.548239    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b04a22a-0651-49fe-9d44-7c0067c2ad14-istiod-ca-cert\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"
Oct 30 17:01:52.548423 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.548269    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b04a22a-0651-49fe-9d44-7c0067c2ad14-workload-certs\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"
Oct 30 17:01:52.548423 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.548313    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b04a22a-0651-49fe-9d44-7c0067c2ad14-istio-envoy\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"
Oct 30 17:01:52.548423 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.548354    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nknwp\" (UniqueName: \"kubernetes.io/projected/5b04a22a-0651-49fe-9d44-7c0067c2ad14-kube-api-access-nknwp\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"
Oct 30 17:01:52.548674 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.548463    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b04a22a-0651-49fe-9d44-7c0067c2ad14-istio-token\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"
Oct 30 17:01:52.548674 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.548560    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b04a22a-0651-49fe-9d44-7c0067c2ad14-istio-data\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"
Oct 30 17:01:52.548674 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.548562    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b04a22a-0651-49fe-9d44-7c0067c2ad14-workload-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"
Oct 30 17:01:52.549065 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.549042    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b04a22a-0651-49fe-9d44-7c0067c2ad14-istiod-ca-cert\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"
Oct 30 17:01:52.549065 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.549052    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b04a22a-0651-49fe-9d44-7c0067c2ad14-credential-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"
Oct 30 17:01:52.549226 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.549128    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b04a22a-0651-49fe-9d44-7c0067c2ad14-workload-certs\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"
Oct 30 17:01:52.549347 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.549332    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b04a22a-0651-49fe-9d44-7c0067c2ad14-istio-data\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"
Oct 30 17:01:52.577431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.571771    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b04a22a-0651-49fe-9d44-7c0067c2ad14-istio-envoy\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"
Oct 30 17:01:52.577431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.572342    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b04a22a-0651-49fe-9d44-7c0067c2ad14-istio-podinfo\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"
Oct 30 17:01:52.578355 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.578305    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nknwp\" (UniqueName: \"kubernetes.io/projected/5b04a22a-0651-49fe-9d44-7c0067c2ad14-kube-api-access-nknwp\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"
Oct 30 17:01:52.580242 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.580212    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b04a22a-0651-49fe-9d44-7c0067c2ad14-istio-token\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"
Oct 30 17:01:52.636158 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.636121    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"
Oct 30 17:01:52.700972 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.700520    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"]
Oct 30 17:01:52.718577 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.715518    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:01:52.718577 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.715612    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:01:52.718577 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.715656    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:01:52.995200 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.995155    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49" event={"ID":"df7cd3d7-ced4-415b-b6c8-eb31a545daac","Type":"ContainerStarted","Data":"9e0a905a9af9d908dd5db3983ede28c64435be767404fe12d61fc5fd11ed4e95"}
Oct 30 17:01:52.996378 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.995277    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49" event={"ID":"df7cd3d7-ced4-415b-b6c8-eb31a545daac","Type":"ContainerStarted","Data":"98ceba1597476f87e15300ecc411eff5cb1fb13e0abcf0840ad7824c1683c328"}
Oct 30 17:01:52.999569 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:52.999505    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"]
Oct 30 17:01:53.001341 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:01:53.001314    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b04a22a_0651_49fe_9d44_7c0067c2ad14.slice/crio-2cac68823eeede402ae71594a60d3e32e8173c667f07a8770393a122b57f01b0 WatchSource:0}: Error finding container 2cac68823eeede402ae71594a60d3e32e8173c667f07a8770393a122b57f01b0: Status 404 returned error can't find the container with id 2cac68823eeede402ae71594a60d3e32e8173c667f07a8770393a122b57f01b0
Oct 30 17:01:53.004170 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:53.004137    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:01:53.004270 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:53.004214    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:01:53.004270 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:53.004257    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:01:53.040792 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:53.040736    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49" podStartSLOduration=2.040716529 podStartE2EDuration="2.040716529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:53.039727886 +0000 UTC m=+889.474630094" watchObservedRunningTime="2026-10-30 17:01:53.040716529 +0000 UTC m=+889.475618735"
Oct 30 17:01:53.187959 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:53.187507    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n9gd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 17:01:53.187959 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:53.187571    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9" podUID="c986e3ae-ffab-420f-92ca-8234d2d918f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 17:01:53.329036 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:53.328954    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"
Oct 30 17:01:53.330744 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:53.330719    4433 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 17:01:53.330848 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:53.330766    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49" podUID="df7cd3d7-ced4-415b-b6c8-eb31a545daac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 17:01:54.039944 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:54.039731    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm" event={"ID":"5b04a22a-0651-49fe-9d44-7c0067c2ad14","Type":"ContainerStarted","Data":"b462e2478aa9042db5e06f2b1a383ea7f4f79460671163bb4104fdd809203b85"}
Oct 30 17:01:54.039944 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:54.039795    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm" event={"ID":"5b04a22a-0651-49fe-9d44-7c0067c2ad14","Type":"ContainerStarted","Data":"2cac68823eeede402ae71594a60d3e32e8173c667f07a8770393a122b57f01b0"}
Oct 30 17:01:54.150276 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:54.150217    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm" podStartSLOduration=2.150198335 podStartE2EDuration="2.150198335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:54.138574819 +0000 UTC m=+890.573477051" watchObservedRunningTime="2026-10-30 17:01:54.150198335 +0000 UTC m=+890.585100541"
Oct 30 17:01:54.187429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:54.186617    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n9gd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 17:01:54.187429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:54.186683    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9" podUID="c986e3ae-ffab-420f-92ca-8234d2d918f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 17:01:54.338660 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:54.329398    4433 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 17:01:54.338660 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:54.329476    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49" podUID="df7cd3d7-ced4-415b-b6c8-eb31a545daac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 17:01:54.641484 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:54.641380    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"
Oct 30 17:01:54.642428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:54.642384    4433 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 17:01:54.642631 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:54.642607    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm" podUID="5b04a22a-0651-49fe-9d44-7c0067c2ad14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 17:01:55.191429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:55.187391    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n9gd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 17:01:55.191429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:55.187470    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9" podUID="c986e3ae-ffab-420f-92ca-8234d2d918f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 17:01:55.329425 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:55.328879    4433 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 17:01:55.329425 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:55.328955    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49" podUID="df7cd3d7-ced4-415b-b6c8-eb31a545daac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 17:01:55.457483 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:55.456865    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:01:55.550431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:55.549510    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 17:01:55.641433 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:55.638813    4433 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 17:01:55.641433 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:55.638877    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm" podUID="5b04a22a-0651-49fe-9d44-7c0067c2ad14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 17:01:55.923500 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:55.923374    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 17:01:56.060223 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:56.060179    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 30 17:01:56.106427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:56.105504    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:01:56.128590 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:56.128523    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:01:56.184041 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:56.184001    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n9gd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 17:01:56.184371 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:56.184323    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9" podUID="c986e3ae-ffab-420f-92ca-8234d2d918f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 17:01:56.192560 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:56.192511    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:01:56.331468 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:56.328853    4433 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 17:01:56.331468 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:56.328918    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49" podUID="df7cd3d7-ced4-415b-b6c8-eb31a545daac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 17:01:56.543485 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:56.540564    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 17:01:56.543485 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:56.541228    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:01:56.574437 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:56.572495    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:01:56.614497 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:56.578568    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:01:56.715662 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:56.696315    4433 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 17:01:56.715662 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:56.696377    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm" podUID="5b04a22a-0651-49fe-9d44-7c0067c2ad14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 17:01:57.183431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:57.147529    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:01:57.183431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:57.147989    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 30 17:01:57.186668 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:57.186635    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 17:01:57.205990 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:57.188230    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 17:01:57.474493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:57.201918    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 17:01:57.474493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:57.329624    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm" podUID="bf43bf3d-f126-40a1-a24f-10214a85ceb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 30 17:01:57.474493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:57.332101    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n9gd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 17:01:57.474493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:57.458204    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9" podUID="c986e3ae-ffab-420f-92ca-8234d2d918f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 17:01:57.474493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:57.338357    4433 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 17:01:57.474493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:57.458269    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49" podUID="df7cd3d7-ced4-415b-b6c8-eb31a545daac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 17:01:57.571446 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:57.504622    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:01:57.620453 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:57.557783    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 17:01:57.639466 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:57.593447    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 17:01:57.639466 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:57.604588    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 17:01:57.666441 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:57.665129    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:01:57.666441 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:57.665225    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 17:01:57.666441 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:57.665264    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 17:01:57.670493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:57.669317    4433 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 17:01:57.670493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:57.669667    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm" podUID="5b04a22a-0651-49fe-9d44-7c0067c2ad14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 17:01:57.830487 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:01:57.805592    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:01:57.838957 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:57.838855    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 17:01:57.848393 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:57.848353    4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:01:57.862860 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:57.862810    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 17:01:57.866724 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:57.866699    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 17:01:57.918253 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:57.918214    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:01:57.949480 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:57.945711    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:01:57.949480 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:57.946647    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:01:58.030102 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:58.030064    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 17:01:58.215556 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:58.215518    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n9gd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 17:01:58.220432 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:58.215818    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9" podUID="c986e3ae-ffab-420f-92ca-8234d2d918f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 17:01:58.333431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:58.328766    4433 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 17:01:58.333431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:58.328837    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49" podUID="df7cd3d7-ced4-415b-b6c8-eb31a545daac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 17:01:58.636870 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:58.636769    4433 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 17:01:58.636870 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:58.636825    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm" podUID="5b04a22a-0651-49fe-9d44-7c0067c2ad14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 17:01:59.181849 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:59.181806    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n9gd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 17:01:59.185425 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:59.182128    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9" podUID="c986e3ae-ffab-420f-92ca-8234d2d918f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 17:01:59.330829 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:59.330790    4433 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 17:01:59.331161 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:59.331130    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49" podUID="df7cd3d7-ced4-415b-b6c8-eb31a545daac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 17:01:59.637205 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:59.637121    4433 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 17:01:59.637593 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:01:59.637199    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm" podUID="5b04a22a-0651-49fe-9d44-7c0067c2ad14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 17:02:00.329725 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:00.329684    4433 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 17:02:00.329919 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:00.329756    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49" podUID="df7cd3d7-ced4-415b-b6c8-eb31a545daac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 17:02:00.540318 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:00.540274    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"
Oct 30 17:02:00.540821 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:00.540793    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"
Oct 30 17:02:00.541687 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:00.541670    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9"
Oct 30 17:02:00.637261 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:00.637171    4433 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 17:02:00.637261 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:00.637232    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm" podUID="5b04a22a-0651-49fe-9d44-7c0067c2ad14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 17:02:01.812367 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:01.812326    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"
Oct 30 17:02:01.813528 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:01.813499    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"
Oct 30 17:02:01.814694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:01.814671    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"
Oct 30 17:02:01.954873 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:01.954829    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"
Oct 30 17:02:01.960478 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:01.960453    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"
Oct 30 17:02:01.962045 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:01.962024    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"
Oct 30 17:02:02.947931 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:02.947899    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d584bb3a-rhhmd"]
Oct 30 17:02:02.953318 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:02.953292    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d584bb3a-rhhmd"
Oct 30 17:02:02.970133 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:02.970103    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d584bb3a-rhhmd"]
Oct 30 17:02:02.977561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:02.977530    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:02:02.977742 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:02.977603    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\""
Oct 30 17:02:03.107529 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:03.107495    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51ecb056-722a-4cfd-a1bd-0d1ef679d7c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d584bb3a-rhhmd\" (UID: \"51ecb056-722a-4cfd-a1bd-0d1ef679d7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d584bb3a-rhhmd"
Oct 30 17:02:03.107726 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:03.107572    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51ecb056-722a-4cfd-a1bd-0d1ef679d7c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d584bb3a-rhhmd\" (UID: \"51ecb056-722a-4cfd-a1bd-0d1ef679d7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d584bb3a-rhhmd"
Oct 30 17:02:03.107726 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:03.107685    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qw6p\" (UniqueName: \"kubernetes.io/projected/51ecb056-722a-4cfd-a1bd-0d1ef679d7c3-kube-api-access-8qw6p\") pod \"maas-api-revoke-sub-d584bb3a-rhhmd\" (UID: \"51ecb056-722a-4cfd-a1bd-0d1ef679d7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d584bb3a-rhhmd"
Oct 30 17:02:03.208944 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:03.208848    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51ecb056-722a-4cfd-a1bd-0d1ef679d7c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d584bb3a-rhhmd\" (UID: \"51ecb056-722a-4cfd-a1bd-0d1ef679d7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d584bb3a-rhhmd"
Oct 30 17:02:03.209094 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:03.208971    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51ecb056-722a-4cfd-a1bd-0d1ef679d7c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d584bb3a-rhhmd\" (UID: \"51ecb056-722a-4cfd-a1bd-0d1ef679d7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d584bb3a-rhhmd"
Oct 30 17:02:03.209094 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:03.209056    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qw6p\" (UniqueName: \"kubernetes.io/projected/51ecb056-722a-4cfd-a1bd-0d1ef679d7c3-kube-api-access-8qw6p\") pod \"maas-api-revoke-sub-d584bb3a-rhhmd\" (UID: \"51ecb056-722a-4cfd-a1bd-0d1ef679d7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d584bb3a-rhhmd"
Oct 30 17:02:03.209676 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:03.209654    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51ecb056-722a-4cfd-a1bd-0d1ef679d7c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d584bb3a-rhhmd\" (UID: \"51ecb056-722a-4cfd-a1bd-0d1ef679d7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d584bb3a-rhhmd"
Oct 30 17:02:03.211228 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:03.211211    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51ecb056-722a-4cfd-a1bd-0d1ef679d7c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d584bb3a-rhhmd\" (UID: \"51ecb056-722a-4cfd-a1bd-0d1ef679d7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d584bb3a-rhhmd"
Oct 30 17:02:03.234198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:03.234170    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qw6p\" (UniqueName: \"kubernetes.io/projected/51ecb056-722a-4cfd-a1bd-0d1ef679d7c3-kube-api-access-8qw6p\") pod \"maas-api-revoke-sub-d584bb3a-rhhmd\" (UID: \"51ecb056-722a-4cfd-a1bd-0d1ef679d7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d584bb3a-rhhmd"
Oct 30 17:02:03.268496 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:03.268459    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d584bb3a-rhhmd"
Oct 30 17:02:03.567341 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:03.567251    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d584bb3a-rhhmd"]
Oct 30 17:02:03.569981 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:02:03.569950    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51ecb056_722a_4cfd_a1bd_0d1ef679d7c3.slice/crio-9a68aab077ccba871101d6c32308c829c6cf86dd10edf1864fe297eca3e6c47a WatchSource:0}: Error finding container 9a68aab077ccba871101d6c32308c829c6cf86dd10edf1864fe297eca3e6c47a: Status 404 returned error can't find the container with id 9a68aab077ccba871101d6c32308c829c6cf86dd10edf1864fe297eca3e6c47a
Oct 30 17:02:03.770615 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:03.770576    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d584bb3a-rhhmd" event={"ID":"51ecb056-722a-4cfd-a1bd-0d1ef679d7c3","Type":"ContainerStarted","Data":"9a68aab077ccba871101d6c32308c829c6cf86dd10edf1864fe297eca3e6c47a"}
Oct 30 17:02:04.778111 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:04.778026    4433 generic.go:356] "Generic (PLEG): container finished" podID="51ecb056-722a-4cfd-a1bd-0d1ef679d7c3" containerID="b20b0434a6779f90a476be2d840035542b4d48f92c58e70c41d16a8811adf19d" exitCode=0
Oct 30 17:02:04.778516 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:04.778104    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d584bb3a-rhhmd" event={"ID":"51ecb056-722a-4cfd-a1bd-0d1ef679d7c3","Type":"ContainerDied","Data":"b20b0434a6779f90a476be2d840035542b4d48f92c58e70c41d16a8811adf19d"}
Oct 30 17:02:05.945190 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:05.945161    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d584bb3a-rhhmd"
Oct 30 17:02:06.010001 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:06.009966    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:02:06.051568 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:06.051476    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51ecb056-722a-4cfd-a1bd-0d1ef679d7c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51ecb056-722a-4cfd-a1bd-0d1ef679d7c3-maas-api-openshift-ca\") pod \"51ecb056-722a-4cfd-a1bd-0d1ef679d7c3\" (UID: \"51ecb056-722a-4cfd-a1bd-0d1ef679d7c3\") "
Oct 30 17:02:06.051781 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:06.051759    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51ecb056-722a-4cfd-a1bd-0d1ef679d7c3-kube-api-access-8qw6p\" (UniqueName: \"kubernetes.io/projected/51ecb056-722a-4cfd-a1bd-0d1ef679d7c3-kube-api-access-8qw6p\") pod \"51ecb056-722a-4cfd-a1bd-0d1ef679d7c3\" (UID: \"51ecb056-722a-4cfd-a1bd-0d1ef679d7c3\") "
Oct 30 17:02:06.051917 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:06.051897    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51ecb056-722a-4cfd-a1bd-0d1ef679d7c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51ecb056-722a-4cfd-a1bd-0d1ef679d7c3-maas-api-xks-ca\") pod \"51ecb056-722a-4cfd-a1bd-0d1ef679d7c3\" (UID: \"51ecb056-722a-4cfd-a1bd-0d1ef679d7c3\") "
Oct 30 17:02:06.052016 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:06.051991    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51ecb056-722a-4cfd-a1bd-0d1ef679d7c3-maas-api-openshift-ca" pod "51ecb056-722a-4cfd-a1bd-0d1ef679d7c3" (UID: "51ecb056-722a-4cfd-a1bd-0d1ef679d7c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:02:06.052288 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:06.052273    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51ecb056-722a-4cfd-a1bd-0d1ef679d7c3-maas-api-openshift-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:02:06.053798 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:06.053771    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51ecb056-722a-4cfd-a1bd-0d1ef679d7c3-kube-api-access-8qw6p" pod "51ecb056-722a-4cfd-a1bd-0d1ef679d7c3" (UID: "51ecb056-722a-4cfd-a1bd-0d1ef679d7c3"). InnerVolumeSpecName "kube-api-access-8qw6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:02:06.053957 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:06.053934    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51ecb056-722a-4cfd-a1bd-0d1ef679d7c3-maas-api-xks-ca" pod "51ecb056-722a-4cfd-a1bd-0d1ef679d7c3" (UID: "51ecb056-722a-4cfd-a1bd-0d1ef679d7c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:02:06.153302 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:06.153261    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qw6p\" (UniqueName: \"kubernetes.io/projected/51ecb056-722a-4cfd-a1bd-0d1ef679d7c3-kube-api-access-8qw6p\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:02:06.153302 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:06.153299    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51ecb056-722a-4cfd-a1bd-0d1ef679d7c3-maas-api-xks-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:02:06.790250 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:06.790207    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d584bb3a-rhhmd" event={"ID":"51ecb056-722a-4cfd-a1bd-0d1ef679d7c3","Type":"ContainerDied","Data":"9a68aab077ccba871101d6c32308c829c6cf86dd10edf1864fe297eca3e6c47a"}
Oct 30 17:02:06.790878 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:06.790268    4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a68aab077ccba871101d6c32308c829c6cf86dd10edf1864fe297eca3e6c47a"
Oct 30 17:02:06.790878 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:06.790235    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d584bb3a-rhhmd"
Oct 30 17:02:07.816456 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:07.816419    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"]
Oct 30 17:02:07.817705 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:07.817674    4433 cpu_manager.go:414] "containerMap: removing container" podUID="51ecb056-722a-4cfd-a1bd-0d1ef679d7c3" containerName="revoke-keys"
Oct 30 17:02:07.817705 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:07.817700    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51ecb056-722a-4cfd-a1bd-0d1ef679d7c3" containerName="revoke-keys"
Oct 30 17:02:07.817864 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:07.817810    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="51ecb056-722a-4cfd-a1bd-0d1ef679d7c3" containerName="revoke-keys"
Oct 30 17:02:07.822348 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:07.822329    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"
Oct 30 17:02:07.840881 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:07.840844    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"]
Oct 30 17:02:07.847476 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:07.847391    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-04248e-openshift-default-dockercfg-bbxjd\""
Oct 30 17:02:07.902697 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:07.902659    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-credential-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"
Oct 30 17:02:07.903260 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:07.903227    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-istio-token\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"
Oct 30 17:02:07.903550 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:07.903521    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-workload-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"
Oct 30 17:02:07.903649 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:07.903575    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-istiod-ca-cert\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"
Oct 30 17:02:07.903649 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:07.903623    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v84ml\" (UniqueName: \"kubernetes.io/projected/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-kube-api-access-v84ml\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"
Oct 30 17:02:07.903774 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:07.903655    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-workload-certs\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"
Oct 30 17:02:07.903774 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:07.903682    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-istio-data\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"
Oct 30 17:02:07.903774 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:07.903730    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-istio-envoy\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"
Oct 30 17:02:07.903956 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:07.903859    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-istio-podinfo\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"
Oct 30 17:02:08.006717 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:08.006672    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-workload-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"
Oct 30 17:02:08.006981 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:08.006961    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-istiod-ca-cert\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"
Oct 30 17:02:08.007122 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:08.007107    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v84ml\" (UniqueName: \"kubernetes.io/projected/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-kube-api-access-v84ml\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"
Oct 30 17:02:08.007234 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:08.007220    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-workload-certs\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"
Oct 30 17:02:08.007335 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:08.007322    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-istio-data\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"
Oct 30 17:02:08.007492 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:08.007473    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-istio-envoy\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"
Oct 30 17:02:08.007714 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:08.007698    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-istio-podinfo\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"
Oct 30 17:02:08.007906 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:08.007890    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-credential-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"
Oct 30 17:02:08.008021 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:08.008005    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-istio-token\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"
Oct 30 17:02:08.017130 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:08.017096    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-workload-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"
Oct 30 17:02:08.018162 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:08.018137    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-istio-data\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"
Oct 30 17:02:08.019347 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:08.018717    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-workload-certs\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"
Oct 30 17:02:08.019500 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:08.018739    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-credential-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"
Oct 30 17:02:08.019591 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:08.019298    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-istiod-ca-cert\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"
Oct 30 17:02:08.022693 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:08.022188    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-istio-podinfo\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"
Oct 30 17:02:08.029232 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:08.029101    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-istio-envoy\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"
Oct 30 17:02:08.055395 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:08.054177    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v84ml\" (UniqueName: \"kubernetes.io/projected/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-kube-api-access-v84ml\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"
Oct 30 17:02:08.055395 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:08.054686    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-istio-token\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"
Oct 30 17:02:08.145356 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:08.145255    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"
Oct 30 17:02:08.513144 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:08.513090    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"]
Oct 30 17:02:08.522444 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:02:08.521777    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21dc53b6_74d2_48a6_b5dd_3b1d2940e845.slice/crio-396b4989f2800e556596fc4c9ef8e59e3a62b1fa79884e0378e1e43c53ec93c4 WatchSource:0}: Error finding container 396b4989f2800e556596fc4c9ef8e59e3a62b1fa79884e0378e1e43c53ec93c4: Status 404 returned error can't find the container with id 396b4989f2800e556596fc4c9ef8e59e3a62b1fa79884e0378e1e43c53ec93c4
Oct 30 17:02:08.524789 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:08.524755    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:02:08.524859 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:08.524826    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:02:08.524859 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:08.524856    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:02:08.823813 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:08.823482    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7" event={"ID":"21dc53b6-74d2-48a6-b5dd-3b1d2940e845","Type":"ContainerStarted","Data":"244f946d18f2dc54e9e26f3fd19380cfc225c9862e2096f4fda2778eda487017"}
Oct 30 17:02:08.823813 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:08.823553    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7" event={"ID":"21dc53b6-74d2-48a6-b5dd-3b1d2940e845","Type":"ContainerStarted","Data":"396b4989f2800e556596fc4c9ef8e59e3a62b1fa79884e0378e1e43c53ec93c4"}
Oct 30 17:02:09.145971 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:09.145869    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"
Oct 30 17:02:09.147562 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:09.147530    4433 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 17:02:09.147719 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:09.147600    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7" podUID="21dc53b6-74d2-48a6-b5dd-3b1d2940e845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 17:02:10.146068 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:10.146034    4433 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 17:02:10.146511 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:10.146097    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7" podUID="21dc53b6-74d2-48a6-b5dd-3b1d2940e845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 17:02:11.146532 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:11.146498    4433 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 17:02:11.146950 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:11.146562    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7" podUID="21dc53b6-74d2-48a6-b5dd-3b1d2940e845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 17:02:12.145979 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:12.145941    4433 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 17:02:12.146194 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:12.145999    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7" podUID="21dc53b6-74d2-48a6-b5dd-3b1d2940e845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 17:02:13.146511 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:13.146474    4433 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 17:02:13.146948 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:13.146532    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7" podUID="21dc53b6-74d2-48a6-b5dd-3b1d2940e845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 17:02:13.683037 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:13.682969    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7" podStartSLOduration=6.682949889 podStartE2EDuration="6.682949889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:08.888805998 +0000 UTC m=+905.323708205" watchObservedRunningTime="2026-10-30 17:02:13.682949889 +0000 UTC m=+910.117852097"
Oct 30 17:02:13.687177 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:13.687126    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"]
Oct 30 17:02:13.700432 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:13.698239    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:02:13.720776 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:13.720734    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-d384dc\"/\"distinct-2-w6-d384dc-kserve-self-signed-certs\""
Oct 30 17:02:13.727053 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:13.727018    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"]
Oct 30 17:02:13.812956 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:13.812915    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/458eba2e-eb03-452a-bcf4-a5e91da83acf-tls-certs\") pod \"distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj\" (UID: \"458eba2e-eb03-452a-bcf4-a5e91da83acf\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:02:13.813133 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:13.812984    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/458eba2e-eb03-452a-bcf4-a5e91da83acf-model-cache\") pod \"distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj\" (UID: \"458eba2e-eb03-452a-bcf4-a5e91da83acf\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:02:13.813133 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:13.813079    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/458eba2e-eb03-452a-bcf4-a5e91da83acf-tmp-dir\") pod \"distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj\" (UID: \"458eba2e-eb03-452a-bcf4-a5e91da83acf\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:02:13.813133 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:13.813104    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/458eba2e-eb03-452a-bcf4-a5e91da83acf-home\") pod \"distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj\" (UID: \"458eba2e-eb03-452a-bcf4-a5e91da83acf\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:02:13.813277 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:13.813137    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/458eba2e-eb03-452a-bcf4-a5e91da83acf-dshm\") pod \"distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj\" (UID: \"458eba2e-eb03-452a-bcf4-a5e91da83acf\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:02:13.813312 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:13.813283    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88hvm\" (UniqueName: \"kubernetes.io/projected/458eba2e-eb03-452a-bcf4-a5e91da83acf-kube-api-access-88hvm\") pod \"distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj\" (UID: \"458eba2e-eb03-452a-bcf4-a5e91da83acf\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:02:13.915221 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:13.915171    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/458eba2e-eb03-452a-bcf4-a5e91da83acf-model-cache\") pod \"distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj\" (UID: \"458eba2e-eb03-452a-bcf4-a5e91da83acf\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:02:13.915396 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:13.915247    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/458eba2e-eb03-452a-bcf4-a5e91da83acf-tmp-dir\") pod \"distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj\" (UID: \"458eba2e-eb03-452a-bcf4-a5e91da83acf\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:02:13.915396 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:13.915276    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/458eba2e-eb03-452a-bcf4-a5e91da83acf-home\") pod \"distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj\" (UID: \"458eba2e-eb03-452a-bcf4-a5e91da83acf\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:02:13.915396 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:13.915321    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/458eba2e-eb03-452a-bcf4-a5e91da83acf-dshm\") pod \"distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj\" (UID: \"458eba2e-eb03-452a-bcf4-a5e91da83acf\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:02:13.915654 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:13.915453    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88hvm\" (UniqueName: \"kubernetes.io/projected/458eba2e-eb03-452a-bcf4-a5e91da83acf-kube-api-access-88hvm\") pod \"distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj\" (UID: \"458eba2e-eb03-452a-bcf4-a5e91da83acf\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:02:13.915654 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:13.915618    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/458eba2e-eb03-452a-bcf4-a5e91da83acf-tls-certs\") pod \"distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj\" (UID: \"458eba2e-eb03-452a-bcf4-a5e91da83acf\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:02:13.916933 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:13.916876    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/458eba2e-eb03-452a-bcf4-a5e91da83acf-tmp-dir\") pod \"distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj\" (UID: \"458eba2e-eb03-452a-bcf4-a5e91da83acf\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:02:13.917654 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:13.917630    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/458eba2e-eb03-452a-bcf4-a5e91da83acf-model-cache\") pod \"distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj\" (UID: \"458eba2e-eb03-452a-bcf4-a5e91da83acf\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:02:13.917926 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:13.917909    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/458eba2e-eb03-452a-bcf4-a5e91da83acf-home\") pod \"distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj\" (UID: \"458eba2e-eb03-452a-bcf4-a5e91da83acf\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:02:13.932426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:13.919217    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/458eba2e-eb03-452a-bcf4-a5e91da83acf-tls-certs\") pod \"distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj\" (UID: \"458eba2e-eb03-452a-bcf4-a5e91da83acf\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:02:13.933004 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:13.932939    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/458eba2e-eb03-452a-bcf4-a5e91da83acf-dshm\") pod \"distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj\" (UID: \"458eba2e-eb03-452a-bcf4-a5e91da83acf\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:02:13.953528 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:13.952904    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88hvm\" (UniqueName: \"kubernetes.io/projected/458eba2e-eb03-452a-bcf4-a5e91da83acf-kube-api-access-88hvm\") pod \"distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj\" (UID: \"458eba2e-eb03-452a-bcf4-a5e91da83acf\") " pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:02:14.026315 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:14.026280    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:02:14.149895 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:14.146631    4433 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 17:02:14.149895 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:14.146689    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7" podUID="21dc53b6-74d2-48a6-b5dd-3b1d2940e845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 17:02:14.381720 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:14.381515    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"]
Oct 30 17:02:14.384018 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:02:14.383988    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod458eba2e_eb03_452a_bcf4_a5e91da83acf.slice/crio-6327cfaa38a16aadd30c0e8b707596dc505806df73406f4b6762baf427833469 WatchSource:0}: Error finding container 6327cfaa38a16aadd30c0e8b707596dc505806df73406f4b6762baf427833469: Status 404 returned error can't find the container with id 6327cfaa38a16aadd30c0e8b707596dc505806df73406f4b6762baf427833469
Oct 30 17:02:14.862114 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:14.862082    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" event={"ID":"458eba2e-eb03-452a-bcf4-a5e91da83acf","Type":"ContainerStarted","Data":"8f562d015630ec70355311e02425fafc2d096fd7dc16a1e5ac07b7dc5c1100b7"}
Oct 30 17:02:14.862306 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:14.862127    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" event={"ID":"458eba2e-eb03-452a-bcf4-a5e91da83acf","Type":"ContainerStarted","Data":"6327cfaa38a16aadd30c0e8b707596dc505806df73406f4b6762baf427833469"}
Oct 30 17:02:14.903232 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:14.903170    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" podStartSLOduration=1.903154276 podStartE2EDuration="1.903154276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:14.902508844 +0000 UTC m=+911.337411050" watchObservedRunningTime="2026-10-30 17:02:14.903154276 +0000 UTC m=+911.338056483"
Oct 30 17:02:15.026621 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:15.026579    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:02:15.146110 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:15.146026    4433 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 17:02:15.146110 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:15.146085    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7" podUID="21dc53b6-74d2-48a6-b5dd-3b1d2940e845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 17:02:16.150706 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:16.150674    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"
Oct 30 17:02:16.151096 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:16.150957    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"
Oct 30 17:02:16.151852 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:16.151829    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"
Oct 30 17:02:22.116648 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.116610    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"]
Oct 30 17:02:22.120796 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.120772    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"
Oct 30 17:02:22.146985 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.146954    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"]
Oct 30 17:02:22.240842 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.240707    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-istio-podinfo\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"
Oct 30 17:02:22.241330 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.241299    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4h8h\" (UniqueName: \"kubernetes.io/projected/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-kube-api-access-d4h8h\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"
Oct 30 17:02:22.241696 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.241673    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-istiod-ca-cert\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"
Oct 30 17:02:22.242026 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.242006    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-istio-token\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"
Oct 30 17:02:22.242231 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.242213    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-istio-envoy\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"
Oct 30 17:02:22.242421 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.242385    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-workload-certs\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"
Oct 30 17:02:22.266169 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.254492    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-istio-data\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"
Oct 30 17:02:22.266169 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.254600    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-credential-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"
Oct 30 17:02:22.266169 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.254655    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-workload-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"
Oct 30 17:02:22.356017 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.355979    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-credential-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"
Oct 30 17:02:22.356236 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.356036    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-workload-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"
Oct 30 17:02:22.356236 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.356097    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-istio-podinfo\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"
Oct 30 17:02:22.356236 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.356164    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4h8h\" (UniqueName: \"kubernetes.io/projected/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-kube-api-access-d4h8h\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"
Oct 30 17:02:22.356236 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.356211    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-istiod-ca-cert\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"
Oct 30 17:02:22.356543 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.356246    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-istio-token\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"
Oct 30 17:02:22.356543 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.356311    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-istio-envoy\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"
Oct 30 17:02:22.356543 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.356344    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-workload-certs\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"
Oct 30 17:02:22.356543 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.356475    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-istio-data\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"
Oct 30 17:02:22.356543 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.356490    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-credential-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"
Oct 30 17:02:22.357092 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.357065    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-workload-certs\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"
Oct 30 17:02:22.357311 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.357292    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-workload-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"
Oct 30 17:02:22.357686 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.357665    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-istio-data\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"
Oct 30 17:02:22.358589 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.358564    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-istiod-ca-cert\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"
Oct 30 17:02:22.359563 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.359537    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-istio-envoy\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"
Oct 30 17:02:22.360279 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.360233    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-istio-podinfo\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"
Oct 30 17:02:22.380840 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.380731    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-istio-token\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"
Oct 30 17:02:22.383083 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.383051    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4h8h\" (UniqueName: \"kubernetes.io/projected/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-kube-api-access-d4h8h\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"
Oct 30 17:02:22.442218 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.441916    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"
Oct 30 17:02:22.742127 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.742019    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"]
Oct 30 17:02:22.753990 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.751772    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"
Oct 30 17:02:22.829620 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.828772    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"]
Oct 30 17:02:22.834165 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.834124    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"]
Oct 30 17:02:22.881018 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.880975    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f851311-31f7-477c-a37c-024f63ea7fcf-istio-token\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"
Oct 30 17:02:22.894666 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.894622    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f851311-31f7-477c-a37c-024f63ea7fcf-istio-podinfo\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"
Oct 30 17:02:22.894972 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.894949    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f851311-31f7-477c-a37c-024f63ea7fcf-istio-data\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"
Oct 30 17:02:22.895207 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.895187    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f851311-31f7-477c-a37c-024f63ea7fcf-workload-certs\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"
Oct 30 17:02:22.895345 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.895327    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f851311-31f7-477c-a37c-024f63ea7fcf-workload-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"
Oct 30 17:02:22.898615 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.898583    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f851311-31f7-477c-a37c-024f63ea7fcf-istio-envoy\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"
Oct 30 17:02:22.898869 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.898848    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f851311-31f7-477c-a37c-024f63ea7fcf-credential-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"
Oct 30 17:02:22.898996 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.898978    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f851311-31f7-477c-a37c-024f63ea7fcf-istiod-ca-cert\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"
Oct 30 17:02:22.899132 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.899115    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrpkl\" (UniqueName: \"kubernetes.io/projected/9f851311-31f7-477c-a37c-024f63ea7fcf-kube-api-access-xrpkl\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"
Oct 30 17:02:22.917737 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.914116    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:02:22.917737 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.914222    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:02:22.917737 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:22.914282    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:02:23.011990 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.011900    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"]
Oct 30 17:02:23.028726 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.019563    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f851311-31f7-477c-a37c-024f63ea7fcf-workload-certs\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"
Oct 30 17:02:23.028726 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.019619    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f851311-31f7-477c-a37c-024f63ea7fcf-workload-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"
Oct 30 17:02:23.028726 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.019671    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f851311-31f7-477c-a37c-024f63ea7fcf-istio-envoy\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"
Oct 30 17:02:23.028726 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.019753    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f851311-31f7-477c-a37c-024f63ea7fcf-credential-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"
Oct 30 17:02:23.028726 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.019781    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f851311-31f7-477c-a37c-024f63ea7fcf-istiod-ca-cert\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"
Oct 30 17:02:23.028726 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.019832    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrpkl\" (UniqueName: \"kubernetes.io/projected/9f851311-31f7-477c-a37c-024f63ea7fcf-kube-api-access-xrpkl\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"
Oct 30 17:02:23.028726 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.019903    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f851311-31f7-477c-a37c-024f63ea7fcf-istio-token\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"
Oct 30 17:02:23.028726 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.019946    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f851311-31f7-477c-a37c-024f63ea7fcf-istio-podinfo\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"
Oct 30 17:02:23.028726 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.020035    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f851311-31f7-477c-a37c-024f63ea7fcf-istio-data\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"
Oct 30 17:02:23.028726 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.020596    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f851311-31f7-477c-a37c-024f63ea7fcf-istio-data\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"
Oct 30 17:02:23.028726 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.020867    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f851311-31f7-477c-a37c-024f63ea7fcf-workload-certs\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"
Oct 30 17:02:23.028726 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.021110    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f851311-31f7-477c-a37c-024f63ea7fcf-workload-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"
Oct 30 17:02:23.037452 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.034180    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f851311-31f7-477c-a37c-024f63ea7fcf-credential-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"
Oct 30 17:02:23.037452 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.034860    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f851311-31f7-477c-a37c-024f63ea7fcf-istiod-ca-cert\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"
Oct 30 17:02:23.112669 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.112616    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"
Oct 30 17:02:23.309509 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.269825    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f851311-31f7-477c-a37c-024f63ea7fcf-istio-podinfo\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"
Oct 30 17:02:23.309509 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.269830    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f851311-31f7-477c-a37c-024f63ea7fcf-istio-token\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"
Oct 30 17:02:23.309509 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.271239    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f851311-31f7-477c-a37c-024f63ea7fcf-istio-envoy\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"
Oct 30 17:02:23.309509 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.290963    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"]
Oct 30 17:02:23.340335 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.340283    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp" event={"ID":"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7","Type":"ContainerStarted","Data":"94884f34f91a81fb98447cc92ea53c0d572c2e0c99756156c261f9948b147f4f"}
Oct 30 17:02:23.351455 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.345268    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrpkl\" (UniqueName: \"kubernetes.io/projected/9f851311-31f7-477c-a37c-024f63ea7fcf-kube-api-access-xrpkl\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"
Oct 30 17:02:23.395437 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.394734    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cb95eff-303b-4088-8ec7-24e9defa3b8a-istio-podinfo\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"
Oct 30 17:02:23.395437 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.394813    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cb95eff-303b-4088-8ec7-24e9defa3b8a-workload-certs\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"
Oct 30 17:02:23.395437 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.394906    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb95eff-303b-4088-8ec7-24e9defa3b8a-workload-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"
Oct 30 17:02:23.395437 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.394931    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cb95eff-303b-4088-8ec7-24e9defa3b8a-istio-envoy\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"
Oct 30 17:02:23.395437 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.394958    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cb95eff-303b-4088-8ec7-24e9defa3b8a-istio-data\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"
Oct 30 17:02:23.395437 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.394993    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzhtm\" (UniqueName: \"kubernetes.io/projected/0cb95eff-303b-4088-8ec7-24e9defa3b8a-kube-api-access-rzhtm\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"
Oct 30 17:02:23.395437 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.395059    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb95eff-303b-4088-8ec7-24e9defa3b8a-credential-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"
Oct 30 17:02:23.395437 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.395120    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cb95eff-303b-4088-8ec7-24e9defa3b8a-istio-token\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"
Oct 30 17:02:23.395437 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.395161    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cb95eff-303b-4088-8ec7-24e9defa3b8a-istiod-ca-cert\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"
Oct 30 17:02:23.427549 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.427287    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"
Oct 30 17:02:23.498438 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.496268    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzhtm\" (UniqueName: \"kubernetes.io/projected/0cb95eff-303b-4088-8ec7-24e9defa3b8a-kube-api-access-rzhtm\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"
Oct 30 17:02:23.498438 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.496347    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb95eff-303b-4088-8ec7-24e9defa3b8a-credential-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"
Oct 30 17:02:23.498438 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.496425    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cb95eff-303b-4088-8ec7-24e9defa3b8a-istio-token\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"
Oct 30 17:02:23.498438 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.496468    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cb95eff-303b-4088-8ec7-24e9defa3b8a-istiod-ca-cert\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"
Oct 30 17:02:23.498438 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.496542    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cb95eff-303b-4088-8ec7-24e9defa3b8a-istio-podinfo\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"
Oct 30 17:02:23.498438 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.496602    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cb95eff-303b-4088-8ec7-24e9defa3b8a-workload-certs\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"
Oct 30 17:02:23.498438 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.496707    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb95eff-303b-4088-8ec7-24e9defa3b8a-workload-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"
Oct 30 17:02:23.498438 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.496732    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cb95eff-303b-4088-8ec7-24e9defa3b8a-istio-envoy\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"
Oct 30 17:02:23.498438 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.496773    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cb95eff-303b-4088-8ec7-24e9defa3b8a-istio-data\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"
Oct 30 17:02:23.498438 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.497216    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cb95eff-303b-4088-8ec7-24e9defa3b8a-istio-data\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"
Oct 30 17:02:23.498438 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.497810    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb95eff-303b-4088-8ec7-24e9defa3b8a-credential-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"
Oct 30 17:02:23.499819 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.499652    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cb95eff-303b-4088-8ec7-24e9defa3b8a-istiod-ca-cert\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"
Oct 30 17:02:23.501901 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.501853    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cb95eff-303b-4088-8ec7-24e9defa3b8a-workload-certs\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"
Oct 30 17:02:23.502465 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.502447    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb95eff-303b-4088-8ec7-24e9defa3b8a-workload-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"
Oct 30 17:02:23.531052 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.530977    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cb95eff-303b-4088-8ec7-24e9defa3b8a-istio-podinfo\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"
Oct 30 17:02:23.573456 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.564892    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cb95eff-303b-4088-8ec7-24e9defa3b8a-istio-envoy\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"
Oct 30 17:02:23.574988 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.574941    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cb95eff-303b-4088-8ec7-24e9defa3b8a-istio-token\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"
Oct 30 17:02:23.585350 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.585277    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzhtm\" (UniqueName: \"kubernetes.io/projected/0cb95eff-303b-4088-8ec7-24e9defa3b8a-kube-api-access-rzhtm\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"
Oct 30 17:02:23.643768 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:23.643725    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"
Oct 30 17:02:24.103259 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:24.103225    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:02:24.134633 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:24.134603    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:02:24.136146 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:02:24.136115    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f851311_31f7_477c_a37c_024f63ea7fcf.slice/crio-b5aa664c51cbd0cf3e05ef1f777a4668db5eb44f89b69d4981ae9f20805bedb6 WatchSource:0}: Error finding container b5aa664c51cbd0cf3e05ef1f777a4668db5eb44f89b69d4981ae9f20805bedb6: Status 404 returned error can't find the container with id b5aa664c51cbd0cf3e05ef1f777a4668db5eb44f89b69d4981ae9f20805bedb6
Oct 30 17:02:24.146680 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:24.146654    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"]
Oct 30 17:02:24.147885 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:24.147847    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:02:24.148079 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:24.147938    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:02:24.148079 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:24.147984    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:02:24.271711 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:24.271680    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"]
Oct 30 17:02:24.275035 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:02:24.275004    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cb95eff_303b_4088_8ec7_24e9defa3b8a.slice/crio-91a8fef80ced192354f45439b7896496dd1edf3a05c5d81c69ef5ab5d1a8d643 WatchSource:0}: Error finding container 91a8fef80ced192354f45439b7896496dd1edf3a05c5d81c69ef5ab5d1a8d643: Status 404 returned error can't find the container with id 91a8fef80ced192354f45439b7896496dd1edf3a05c5d81c69ef5ab5d1a8d643
Oct 30 17:02:24.289667 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:24.289046    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:02:24.289667 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:24.289122    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:02:24.289667 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:24.289156    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:02:24.350477 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:24.350440    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d" event={"ID":"9f851311-31f7-477c-a37c-024f63ea7fcf","Type":"ContainerStarted","Data":"e91a0c7cbed7192ffbe2e5fd8edf39d01843d425024883e14867ce49744acbae"}
Oct 30 17:02:24.350922 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:24.350506    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d" event={"ID":"9f851311-31f7-477c-a37c-024f63ea7fcf","Type":"ContainerStarted","Data":"b5aa664c51cbd0cf3e05ef1f777a4668db5eb44f89b69d4981ae9f20805bedb6"}
Oct 30 17:02:24.353533 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:24.353503    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp" event={"ID":"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7","Type":"ContainerStarted","Data":"06f4bf8086cc8a605ea07e6b6360b079494cd179289e0c849ea2eb0c0c62f5c7"}
Oct 30 17:02:24.354902 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:24.354865    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz" event={"ID":"0cb95eff-303b-4088-8ec7-24e9defa3b8a","Type":"ContainerStarted","Data":"91a8fef80ced192354f45439b7896496dd1edf3a05c5d81c69ef5ab5d1a8d643"}
Oct 30 17:02:24.373275 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:24.373242    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:02:24.403905 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:24.403740    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d" podStartSLOduration=2.403720324 podStartE2EDuration="2.403720324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:24.396254602 +0000 UTC m=+920.831156809" watchObservedRunningTime="2026-10-30 17:02:24.403720324 +0000 UTC m=+920.838622532"
Oct 30 17:02:24.429585 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:24.429553    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"
Oct 30 17:02:24.442451 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:24.441617    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp" podStartSLOduration=2.441598705 podStartE2EDuration="2.441598705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:24.441097729 +0000 UTC m=+920.875999936" watchObservedRunningTime="2026-10-30 17:02:24.441598705 +0000 UTC m=+920.876500911"
Oct 30 17:02:24.450338 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:24.450310    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"
Oct 30 17:02:24.450662 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:24.450642    4433 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 17:02:24.450797 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:24.450775    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp" podUID="47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 17:02:25.380430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:25.379587    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz" event={"ID":"0cb95eff-303b-4088-8ec7-24e9defa3b8a","Type":"ContainerStarted","Data":"84e8e015d2fdf6a281533d9fc29e1ebbbe095983550b77fac3ebaceadac11209"}
Oct 30 17:02:25.436427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:25.435943    4433 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 17:02:25.436427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:25.436000    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d" podUID="9f851311-31f7-477c-a37c-024f63ea7fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 17:02:25.445600 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:25.445396    4433 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 17:02:25.445600 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:25.445477    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp" podUID="47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 17:02:25.645572 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:25.645488    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"
Oct 30 17:02:25.646836 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:25.646810    4433 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 17:02:25.646928 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:25.646862    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz" podUID="0cb95eff-303b-4088-8ec7-24e9defa3b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 17:02:26.436246 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:26.435769    4433 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 17:02:26.436246 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:26.435837    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d" podUID="9f851311-31f7-477c-a37c-024f63ea7fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 17:02:26.442956 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:26.442926    4433 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 17:02:26.443107 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:26.442978    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp" podUID="47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 17:02:26.645321 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:26.645277    4433 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 17:02:26.645523 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:26.645336    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz" podUID="0cb95eff-303b-4088-8ec7-24e9defa3b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 17:02:27.085235 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:02:27.085193    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:02:27.430423 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:27.429944    4433 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 17:02:27.430423 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:27.430019    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d" podUID="9f851311-31f7-477c-a37c-024f63ea7fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 17:02:27.443426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:27.443326    4433 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 17:02:27.443426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:27.443387    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp" podUID="47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 17:02:27.650449 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:27.646462    4433 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 17:02:27.650449 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:27.646523    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz" podUID="0cb95eff-303b-4088-8ec7-24e9defa3b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 17:02:28.458982 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:28.458518    4433 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 17:02:28.458982 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:28.458580    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp" podUID="47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 17:02:28.459974 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:28.459949    4433 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 17:02:28.460148 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:28.460124    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d" podUID="9f851311-31f7-477c-a37c-024f63ea7fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 17:02:28.649622 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:28.645819    4433 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 17:02:28.649622 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:28.645880    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz" podUID="0cb95eff-303b-4088-8ec7-24e9defa3b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 17:02:29.430428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:29.429964    4433 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 17:02:29.430428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:29.430039    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d" podUID="9f851311-31f7-477c-a37c-024f63ea7fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 17:02:29.444429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:29.443754    4433 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 17:02:29.444429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:29.443815    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp" podUID="47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 17:02:29.646429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:29.645287    4433 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 17:02:29.646429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:29.645351    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz" podUID="0cb95eff-303b-4088-8ec7-24e9defa3b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 17:02:30.430370 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:30.430326    4433 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 17:02:30.430758 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:30.430722    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d" podUID="9f851311-31f7-477c-a37c-024f63ea7fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 17:02:30.443613 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:30.443581    4433 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 17:02:30.443858 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:30.443822    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp" podUID="47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 17:02:30.645292 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:30.645253    4433 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 17:02:30.645587 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:30.645552    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz" podUID="0cb95eff-303b-4088-8ec7-24e9defa3b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 17:02:31.430428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:31.429924    4433 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 17:02:31.430428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:31.430009    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d" podUID="9f851311-31f7-477c-a37c-024f63ea7fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 17:02:31.645093 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:31.645044    4433 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 17:02:31.645300 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:31.645118    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz" podUID="0cb95eff-303b-4088-8ec7-24e9defa3b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 17:02:31.706550 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:31.706452    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"
Oct 30 17:02:31.707377 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:31.707355    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"
Oct 30 17:02:31.708394 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:31.708373    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"
Oct 30 17:02:31.763396 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:31.763340    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz" podStartSLOduration=9.763322078 podStartE2EDuration="9.763322078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:25.433803562 +0000 UTC m=+921.868705779" watchObservedRunningTime="2026-10-30 17:02:31.763322078 +0000 UTC m=+928.198224286"
Oct 30 17:02:32.435138 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:32.435103    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"
Oct 30 17:02:32.435782 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:32.435531    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"
Oct 30 17:02:32.436376 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:32.436355    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"
Oct 30 17:02:32.649107 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:32.649071    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"
Oct 30 17:02:32.649615 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:32.649590    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"
Oct 30 17:02:32.650559 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:32.650537    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"
Oct 30 17:02:35.475670 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:35.475632    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pp9d"]
Oct 30 17:02:35.480910 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:35.480880    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pp9d"
Oct 30 17:02:35.498723 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:35.498690    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 17:02:35.498723 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:35.498690    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 17:02:35.498723 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:35.498699    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 17:02:35.509744 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:35.509716    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xjmfv\""
Oct 30 17:02:35.515074 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:35.514985    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pp9d"]
Oct 30 17:02:35.582366 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:35.582331    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bf2a3652-49a3-4e6e-b319-562eea16f39e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pp9d\" (UID: \"bf2a3652-49a3-4e6e-b319-562eea16f39e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pp9d"
Oct 30 17:02:35.582551 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:35.582483    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bf2a3652-49a3-4e6e-b319-562eea16f39e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pp9d\" (UID: \"bf2a3652-49a3-4e6e-b319-562eea16f39e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pp9d"
Oct 30 17:02:35.582551 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:35.582516    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bf2a3652-49a3-4e6e-b319-562eea16f39e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pp9d\" (UID: \"bf2a3652-49a3-4e6e-b319-562eea16f39e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pp9d"
Oct 30 17:02:35.582668 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:35.582645    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bf2a3652-49a3-4e6e-b319-562eea16f39e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pp9d\" (UID: \"bf2a3652-49a3-4e6e-b319-562eea16f39e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pp9d"
Oct 30 17:02:35.582700 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:35.582678    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxhhx\" (UniqueName: \"kubernetes.io/projected/bf2a3652-49a3-4e6e-b319-562eea16f39e-kube-api-access-cxhhx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pp9d\" (UID: \"bf2a3652-49a3-4e6e-b319-562eea16f39e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pp9d"
Oct 30 17:02:35.582734 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:35.582708    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf2a3652-49a3-4e6e-b319-562eea16f39e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pp9d\" (UID: \"bf2a3652-49a3-4e6e-b319-562eea16f39e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pp9d"
Oct 30 17:02:35.582767 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:35.582744    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf2a3652-49a3-4e6e-b319-562eea16f39e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pp9d\" (UID: \"bf2a3652-49a3-4e6e-b319-562eea16f39e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pp9d"
Oct 30 17:02:35.683898 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:35.683862    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bf2a3652-49a3-4e6e-b319-562eea16f39e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pp9d\" (UID: \"bf2a3652-49a3-4e6e-b319-562eea16f39e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pp9d"
Oct 30 17:02:35.684076 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:35.683912    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bf2a3652-49a3-4e6e-b319-562eea16f39e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pp9d\" (UID: \"bf2a3652-49a3-4e6e-b319-562eea16f39e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pp9d"
Oct 30 17:02:35.684076 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:35.683980    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bf2a3652-49a3-4e6e-b319-562eea16f39e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pp9d\" (UID: \"bf2a3652-49a3-4e6e-b319-562eea16f39e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pp9d"
Oct 30 17:02:35.684076 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:35.684009    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxhhx\" (UniqueName: \"kubernetes.io/projected/bf2a3652-49a3-4e6e-b319-562eea16f39e-kube-api-access-cxhhx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pp9d\" (UID: \"bf2a3652-49a3-4e6e-b319-562eea16f39e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pp9d"
Oct 30 17:02:35.684076 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:35.684038    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf2a3652-49a3-4e6e-b319-562eea16f39e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pp9d\" (UID: \"bf2a3652-49a3-4e6e-b319-562eea16f39e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pp9d"
Oct 30 17:02:35.684076 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:35.684054    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf2a3652-49a3-4e6e-b319-562eea16f39e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pp9d\" (UID: \"bf2a3652-49a3-4e6e-b319-562eea16f39e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pp9d"
Oct 30 17:02:35.684324 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:35.684144    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bf2a3652-49a3-4e6e-b319-562eea16f39e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pp9d\" (UID: \"bf2a3652-49a3-4e6e-b319-562eea16f39e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pp9d"
Oct 30 17:02:35.685028 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:35.684995    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bf2a3652-49a3-4e6e-b319-562eea16f39e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pp9d\" (UID: \"bf2a3652-49a3-4e6e-b319-562eea16f39e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pp9d"
Oct 30 17:02:35.686492 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:35.686463    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bf2a3652-49a3-4e6e-b319-562eea16f39e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pp9d\" (UID: \"bf2a3652-49a3-4e6e-b319-562eea16f39e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pp9d"
Oct 30 17:02:35.686625 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:35.686537    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bf2a3652-49a3-4e6e-b319-562eea16f39e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pp9d\" (UID: \"bf2a3652-49a3-4e6e-b319-562eea16f39e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pp9d"
Oct 30 17:02:35.686757 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:35.686736    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf2a3652-49a3-4e6e-b319-562eea16f39e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pp9d\" (UID: \"bf2a3652-49a3-4e6e-b319-562eea16f39e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pp9d"
Oct 30 17:02:35.686817 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:35.686775    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bf2a3652-49a3-4e6e-b319-562eea16f39e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pp9d\" (UID: \"bf2a3652-49a3-4e6e-b319-562eea16f39e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pp9d"
Oct 30 17:02:35.708149 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:35.708121    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf2a3652-49a3-4e6e-b319-562eea16f39e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pp9d\" (UID: \"bf2a3652-49a3-4e6e-b319-562eea16f39e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pp9d"
Oct 30 17:02:35.708333 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:35.708314    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxhhx\" (UniqueName: \"kubernetes.io/projected/bf2a3652-49a3-4e6e-b319-562eea16f39e-kube-api-access-cxhhx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pp9d\" (UID: \"bf2a3652-49a3-4e6e-b319-562eea16f39e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pp9d"
Oct 30 17:02:35.793815 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:35.793735    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pp9d"
Oct 30 17:02:36.116197 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:36.116171    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pp9d"]
Oct 30 17:02:36.118737 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:02:36.118707    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf2a3652_49a3_4e6e_b319_562eea16f39e.slice/crio-dc7e16d7d55d094aad2450f5ecb4f3f78511d513f2262a4558c10d050cd0fffa WatchSource:0}: Error finding container dc7e16d7d55d094aad2450f5ecb4f3f78511d513f2262a4558c10d050cd0fffa: Status 404 returned error can't find the container with id dc7e16d7d55d094aad2450f5ecb4f3f78511d513f2262a4558c10d050cd0fffa
Oct 30 17:02:36.543803 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:36.543763    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pp9d" event={"ID":"bf2a3652-49a3-4e6e-b319-562eea16f39e","Type":"ContainerStarted","Data":"dc7e16d7d55d094aad2450f5ecb4f3f78511d513f2262a4558c10d050cd0fffa"}
Oct 30 17:02:38.048436 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:38.048382    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"]
Oct 30 17:02:38.052296 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:38.052269    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"
Oct 30 17:02:38.082395 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:38.082360    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"]
Oct 30 17:02:38.122340 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:38.122286    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-istio-envoy\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"
Oct 30 17:02:38.122574 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:38.122354    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-workload-certs\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"
Oct 30 17:02:38.122574 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:38.122493    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-istio-podinfo\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"
Oct 30 17:02:38.122574 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:38.122558    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-istio-data\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"
Oct 30 17:02:38.122763 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:38.122657    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-istio-token\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"
Oct 30 17:02:38.122763 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:38.122743    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-credential-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"
Oct 30 17:02:38.122870 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:38.122807    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-istiod-ca-cert\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"
Oct 30 17:02:38.122870 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:38.122826    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gfkg\" (UniqueName: \"kubernetes.io/projected/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-kube-api-access-2gfkg\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"
Oct 30 17:02:38.122870 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:38.122862    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-workload-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"
Oct 30 17:02:38.224298 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:38.224258    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-istiod-ca-cert\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"
Oct 30 17:02:38.224518 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:38.224313    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gfkg\" (UniqueName: \"kubernetes.io/projected/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-kube-api-access-2gfkg\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"
Oct 30 17:02:38.224518 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:38.224364    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-workload-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"
Oct 30 17:02:38.224518 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:38.224453    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-istio-envoy\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"
Oct 30 17:02:38.224518 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:38.224481    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-workload-certs\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"
Oct 30 17:02:38.224518 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:38.224513    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-istio-podinfo\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"
Oct 30 17:02:38.224806 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:38.224546    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-istio-data\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"
Oct 30 17:02:38.224806 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:38.224576    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-istio-token\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"
Oct 30 17:02:38.224806 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:38.224638    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-credential-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"
Oct 30 17:02:38.224969 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:38.224816    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-workload-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"
Oct 30 17:02:38.225010 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:38.224971    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-credential-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"
Oct 30 17:02:38.225097 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:38.225065    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-istio-data\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"
Oct 30 17:02:38.225177 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:38.225121    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-istiod-ca-cert\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"
Oct 30 17:02:38.225298 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:38.225281    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-workload-certs\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"
Oct 30 17:02:38.227239 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:38.227210    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-istio-envoy\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"
Oct 30 17:02:38.227703 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:38.227681    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-istio-podinfo\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"
Oct 30 17:02:38.247958 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:38.247926    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-istio-token\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"
Oct 30 17:02:38.248112 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:38.247988    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gfkg\" (UniqueName: \"kubernetes.io/projected/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-kube-api-access-2gfkg\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"
Oct 30 17:02:38.371100 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:38.371013    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"
Oct 30 17:02:38.694395 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:38.694373    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"]
Oct 30 17:02:38.696875 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:02:38.696848    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16eaa2d1_45c5_4b91_8638_e6a5c492de5c.slice/crio-582572f6b8fd9a984ead60e19faf4da3193d98ce0f9e676f55861056c49801cb WatchSource:0}: Error finding container 582572f6b8fd9a984ead60e19faf4da3193d98ce0f9e676f55861056c49801cb: Status 404 returned error can't find the container with id 582572f6b8fd9a984ead60e19faf4da3193d98ce0f9e676f55861056c49801cb
Oct 30 17:02:38.699684 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:38.699649    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:02:38.699801 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:38.699716    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:02:38.699801 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:38.699765    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:02:39.484366 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:39.484322    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:02:39.582286 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:39.582249    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s" event={"ID":"16eaa2d1-45c5-4b91-8638-e6a5c492de5c","Type":"ContainerStarted","Data":"68d3078d10323e47f2fa32f2fbf0cfea638f5553493f1fa8d6de7dbc1b0adb0c"}
Oct 30 17:02:39.582493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:39.582311    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s" event={"ID":"16eaa2d1-45c5-4b91-8638-e6a5c492de5c","Type":"ContainerStarted","Data":"582572f6b8fd9a984ead60e19faf4da3193d98ce0f9e676f55861056c49801cb"}
Oct 30 17:02:39.629421 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:39.629346    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s" podStartSLOduration=1.629327347 podStartE2EDuration="1.629327347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:39.626253912 +0000 UTC m=+936.061156118" watchObservedRunningTime="2026-10-30 17:02:39.629327347 +0000 UTC m=+936.064229553"
Oct 30 17:02:39.662032 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:39.650528    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:02:39.662032 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:39.650604    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:02:40.372428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:40.371552    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"
Oct 30 17:02:40.380427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:40.373667    4433 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 17:02:40.380427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:40.373766    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s" podUID="16eaa2d1-45c5-4b91-8638-e6a5c492de5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 17:02:40.624513 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:40.621685    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pp9d" event={"ID":"bf2a3652-49a3-4e6e-b319-562eea16f39e","Type":"ContainerStarted","Data":"1dc76c5c7979c9fd43a56bab0aff6d20e95a3c95b7236891273392948caf4183"}
Oct 30 17:02:40.624513 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:40.622002    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pp9d"
Oct 30 17:02:40.624513 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:40.623885    4433 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-7pp9d container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:02:40.624513 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:40.623924    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pp9d" podUID="bf2a3652-49a3-4e6e-b319-562eea16f39e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:02:41.376446 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:41.373439    4433 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 17:02:41.376446 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:41.373513    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s" podUID="16eaa2d1-45c5-4b91-8638-e6a5c492de5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 17:02:41.635424 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:41.635317    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pp9d"
Oct 30 17:02:41.726553 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:41.726467    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pp9d" podStartSLOduration=3.1978307360000002 podStartE2EDuration="6.726442118s" totalImagesPullingTime="3.528611382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:40.702813611 +0000 UTC m=+937.137715839" watchObservedRunningTime="2026-10-30 17:02:41.726442118 +0000 UTC m=+938.161344349"
Oct 30 17:02:42.372436 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:42.371877    4433 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 17:02:42.372436 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:42.371936    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s" podUID="16eaa2d1-45c5-4b91-8638-e6a5c492de5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 17:02:43.375820 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:43.375780    4433 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 17:02:43.376375 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:43.375854    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s" podUID="16eaa2d1-45c5-4b91-8638-e6a5c492de5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 17:02:44.376426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:44.375868    4433 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 17:02:44.376426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:44.375939    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s" podUID="16eaa2d1-45c5-4b91-8638-e6a5c492de5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 17:02:45.372208 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:45.372172    4433 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 17:02:45.372399 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:45.372227    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s" podUID="16eaa2d1-45c5-4b91-8638-e6a5c492de5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 17:02:47.367963 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:47.367869    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"
Oct 30 17:02:47.368704 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:47.368677    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"
Oct 30 17:02:47.369963 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:02:47.369935    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"
Oct 30 17:02:56.978713 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:02:56.978676    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:03:14.766950 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:14.766915    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:03:21.324112 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:21.324076    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fe3dc03-bnxpc"]
Oct 30 17:03:21.326881 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:21.326864    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fe3dc03-bnxpc"
Oct 30 17:03:21.341102 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:21.341079    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:03:21.352503 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:21.352474    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\""
Oct 30 17:03:21.356981 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:21.356952    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fe3dc03-bnxpc"]
Oct 30 17:03:21.463162 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:21.463124    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r552f\" (UniqueName: \"kubernetes.io/projected/5a0a0b2b-ee78-42b5-9156-a0c5a261fc03-kube-api-access-r552f\") pod \"maas-api-revoke-sub-8fe3dc03-bnxpc\" (UID: \"5a0a0b2b-ee78-42b5-9156-a0c5a261fc03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fe3dc03-bnxpc"
Oct 30 17:03:21.463344 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:21.463269    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a0a0b2b-ee78-42b5-9156-a0c5a261fc03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fe3dc03-bnxpc\" (UID: \"5a0a0b2b-ee78-42b5-9156-a0c5a261fc03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fe3dc03-bnxpc"
Oct 30 17:03:21.463471 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:21.463450    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a0a0b2b-ee78-42b5-9156-a0c5a261fc03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fe3dc03-bnxpc\" (UID: \"5a0a0b2b-ee78-42b5-9156-a0c5a261fc03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fe3dc03-bnxpc"
Oct 30 17:03:21.564747 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:21.564706    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r552f\" (UniqueName: \"kubernetes.io/projected/5a0a0b2b-ee78-42b5-9156-a0c5a261fc03-kube-api-access-r552f\") pod \"maas-api-revoke-sub-8fe3dc03-bnxpc\" (UID: \"5a0a0b2b-ee78-42b5-9156-a0c5a261fc03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fe3dc03-bnxpc"
Oct 30 17:03:21.564934 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:21.564793    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a0a0b2b-ee78-42b5-9156-a0c5a261fc03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fe3dc03-bnxpc\" (UID: \"5a0a0b2b-ee78-42b5-9156-a0c5a261fc03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fe3dc03-bnxpc"
Oct 30 17:03:21.564934 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:21.564873    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a0a0b2b-ee78-42b5-9156-a0c5a261fc03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fe3dc03-bnxpc\" (UID: \"5a0a0b2b-ee78-42b5-9156-a0c5a261fc03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fe3dc03-bnxpc"
Oct 30 17:03:21.565500 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:21.565467    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a0a0b2b-ee78-42b5-9156-a0c5a261fc03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fe3dc03-bnxpc\" (UID: \"5a0a0b2b-ee78-42b5-9156-a0c5a261fc03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fe3dc03-bnxpc"
Oct 30 17:03:21.567229 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:21.567211    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a0a0b2b-ee78-42b5-9156-a0c5a261fc03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fe3dc03-bnxpc\" (UID: \"5a0a0b2b-ee78-42b5-9156-a0c5a261fc03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fe3dc03-bnxpc"
Oct 30 17:03:21.590390 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:21.590324    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r552f\" (UniqueName: \"kubernetes.io/projected/5a0a0b2b-ee78-42b5-9156-a0c5a261fc03-kube-api-access-r552f\") pod \"maas-api-revoke-sub-8fe3dc03-bnxpc\" (UID: \"5a0a0b2b-ee78-42b5-9156-a0c5a261fc03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fe3dc03-bnxpc"
Oct 30 17:03:21.639661 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:21.639622    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fe3dc03-bnxpc"
Oct 30 17:03:21.942152 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:21.942127    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fe3dc03-bnxpc"]
Oct 30 17:03:21.944562 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:03:21.944536    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a0a0b2b_ee78_42b5_9156_a0c5a261fc03.slice/crio-624bfa467c0d3b8dfeeb7ef51d842f3082fca8fb887e28f4f69a23febafec810 WatchSource:0}: Error finding container 624bfa467c0d3b8dfeeb7ef51d842f3082fca8fb887e28f4f69a23febafec810: Status 404 returned error can't find the container with id 624bfa467c0d3b8dfeeb7ef51d842f3082fca8fb887e28f4f69a23febafec810
Oct 30 17:03:21.976803 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:21.976772    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fe3dc03-bnxpc" event={"ID":"5a0a0b2b-ee78-42b5-9156-a0c5a261fc03","Type":"ContainerStarted","Data":"624bfa467c0d3b8dfeeb7ef51d842f3082fca8fb887e28f4f69a23febafec810"}
Oct 30 17:03:22.986031 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:22.985992    4433 generic.go:356] "Generic (PLEG): container finished" podID="5a0a0b2b-ee78-42b5-9156-a0c5a261fc03" containerID="8ec5a69429bcf65692c50fb99431daf08c1b6f6ab6d6866cb43f51b3d7469f18" exitCode=0
Oct 30 17:03:22.986506 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:22.986091    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fe3dc03-bnxpc" event={"ID":"5a0a0b2b-ee78-42b5-9156-a0c5a261fc03","Type":"ContainerDied","Data":"8ec5a69429bcf65692c50fb99431daf08c1b6f6ab6d6866cb43f51b3d7469f18"}
Oct 30 17:03:24.166263 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:24.166232    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fe3dc03-bnxpc"
Oct 30 17:03:24.318904 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:24.318811    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a0a0b2b-ee78-42b5-9156-a0c5a261fc03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a0a0b2b-ee78-42b5-9156-a0c5a261fc03-maas-api-xks-ca\") pod \"5a0a0b2b-ee78-42b5-9156-a0c5a261fc03\" (UID: \"5a0a0b2b-ee78-42b5-9156-a0c5a261fc03\") "
Oct 30 17:03:24.318904 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:24.318864    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a0a0b2b-ee78-42b5-9156-a0c5a261fc03-kube-api-access-r552f\" (UniqueName: \"kubernetes.io/projected/5a0a0b2b-ee78-42b5-9156-a0c5a261fc03-kube-api-access-r552f\") pod \"5a0a0b2b-ee78-42b5-9156-a0c5a261fc03\" (UID: \"5a0a0b2b-ee78-42b5-9156-a0c5a261fc03\") "
Oct 30 17:03:24.319189 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:24.318923    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a0a0b2b-ee78-42b5-9156-a0c5a261fc03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a0a0b2b-ee78-42b5-9156-a0c5a261fc03-maas-api-openshift-ca\") pod \"5a0a0b2b-ee78-42b5-9156-a0c5a261fc03\" (UID: \"5a0a0b2b-ee78-42b5-9156-a0c5a261fc03\") "
Oct 30 17:03:24.319342 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:24.319316    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a0a0b2b-ee78-42b5-9156-a0c5a261fc03-maas-api-openshift-ca" pod "5a0a0b2b-ee78-42b5-9156-a0c5a261fc03" (UID: "5a0a0b2b-ee78-42b5-9156-a0c5a261fc03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:03:24.319493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:24.319472    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a0a0b2b-ee78-42b5-9156-a0c5a261fc03-maas-api-openshift-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:03:24.321040 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:24.321019    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a0a0b2b-ee78-42b5-9156-a0c5a261fc03-maas-api-xks-ca" pod "5a0a0b2b-ee78-42b5-9156-a0c5a261fc03" (UID: "5a0a0b2b-ee78-42b5-9156-a0c5a261fc03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:03:24.321105 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:24.321063    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a0a0b2b-ee78-42b5-9156-a0c5a261fc03-kube-api-access-r552f" pod "5a0a0b2b-ee78-42b5-9156-a0c5a261fc03" (UID: "5a0a0b2b-ee78-42b5-9156-a0c5a261fc03"). InnerVolumeSpecName "kube-api-access-r552f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:03:24.420985 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:24.420946    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a0a0b2b-ee78-42b5-9156-a0c5a261fc03-maas-api-xks-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:03:24.420985 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:24.420979    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r552f\" (UniqueName: \"kubernetes.io/projected/5a0a0b2b-ee78-42b5-9156-a0c5a261fc03-kube-api-access-r552f\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:03:24.998647 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:24.998616    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fe3dc03-bnxpc"
Oct 30 17:03:24.998647 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:24.998638    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fe3dc03-bnxpc" event={"ID":"5a0a0b2b-ee78-42b5-9156-a0c5a261fc03","Type":"ContainerDied","Data":"624bfa467c0d3b8dfeeb7ef51d842f3082fca8fb887e28f4f69a23febafec810"}
Oct 30 17:03:24.998873 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:24.998695    4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="624bfa467c0d3b8dfeeb7ef51d842f3082fca8fb887e28f4f69a23febafec810"
Oct 30 17:03:26.978611 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:03:26.978574    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:03:32.811552 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:32.811500    4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded"
Oct 30 17:03:32.844548 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:32.844489    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 17:03:32.875570 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:32.875516    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:03:32.876209 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:32.876177    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 17:03:32.939429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:32.928863    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:03:32.970466 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:32.949731    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:03:33.041612 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:33.041558    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:03:33.045065 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:33.045002    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:03:33.358561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:33.358501    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" podUID="458eba2e-eb03-452a-bcf4-a5e91da83acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:03:33.487431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:33.486518    4433 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:03:33.487431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:33.486589    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d" podUID="9f851311-31f7-477c-a37c-024f63ea7fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:03:33.879308 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:33.632167    4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:03:33.883538 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:33.883496    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:03:33.898501 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:33.898455    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 30 17:03:33.921875 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:33.921825    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 17:03:33.972556 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:33.972506    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:03:34.003437 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:33.972990    4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": context deadline exceeded"
Oct 30 17:03:34.079633 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:33.973869    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:03:34.081175 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:34.008696    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 17:03:34.081429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:34.081393    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 17:03:34.081639 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:34.079224    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:03:34.081773 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:34.079160    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 30 17:03:34.081926 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:34.081647    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 17:03:34.084298 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:34.084279    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:03:34.086163 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:34.086145    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:03:34.086375 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:34.086347    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:03:34.086684 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:34.086656    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 17:03:34.223770 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:34.223740    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 17:03:34.255133 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:34.255095    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm" podUID="bf43bf3d-f126-40a1-a24f-10214a85ceb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:03:34.346931 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:34.346905    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:03:34.416699 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:34.416669    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 17:03:34.426757 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:34.426732    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:03:34.433820 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:34.433191    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 17:03:34.496103 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:34.495525    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 17:03:34.506437 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:34.505652    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:03:38.014445 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.014389    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"]
Oct 30 17:03:38.014963 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.014948    4433 cpu_manager.go:414] "containerMap: removing container" podUID="5a0a0b2b-ee78-42b5-9156-a0c5a261fc03" containerName="revoke-keys"
Oct 30 17:03:38.015014 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.014965    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a0a0b2b-ee78-42b5-9156-a0c5a261fc03" containerName="revoke-keys"
Oct 30 17:03:38.015075 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.015066    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a0a0b2b-ee78-42b5-9156-a0c5a261fc03" containerName="revoke-keys"
Oct 30 17:03:38.017907 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.017882    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"
Oct 30 17:03:38.048938 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.048899    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"]
Oct 30 17:03:38.136566 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.136528    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07938ade-58fb-4ca9-a7a2-618a0e318196-workload-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"
Oct 30 17:03:38.136566 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.136569    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07938ade-58fb-4ca9-a7a2-618a0e318196-credential-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"
Oct 30 17:03:38.136786 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.136589    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07938ade-58fb-4ca9-a7a2-618a0e318196-istio-podinfo\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"
Oct 30 17:03:38.136786 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.136657    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q26g9\" (UniqueName: \"kubernetes.io/projected/07938ade-58fb-4ca9-a7a2-618a0e318196-kube-api-access-q26g9\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"
Oct 30 17:03:38.136786 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.136737    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07938ade-58fb-4ca9-a7a2-618a0e318196-istio-envoy\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"
Oct 30 17:03:38.136786 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.136773    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07938ade-58fb-4ca9-a7a2-618a0e318196-istio-token\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"
Oct 30 17:03:38.136912 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.136876    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07938ade-58fb-4ca9-a7a2-618a0e318196-istiod-ca-cert\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"
Oct 30 17:03:38.136997 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.136981    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07938ade-58fb-4ca9-a7a2-618a0e318196-workload-certs\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"
Oct 30 17:03:38.137084 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.137071    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07938ade-58fb-4ca9-a7a2-618a0e318196-istio-data\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"
Oct 30 17:03:38.237856 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.237822    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07938ade-58fb-4ca9-a7a2-618a0e318196-istiod-ca-cert\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"
Oct 30 17:03:38.238051 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.237902    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07938ade-58fb-4ca9-a7a2-618a0e318196-workload-certs\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"
Oct 30 17:03:38.238051 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.237984    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07938ade-58fb-4ca9-a7a2-618a0e318196-istio-data\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"
Oct 30 17:03:38.238156 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.238089    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07938ade-58fb-4ca9-a7a2-618a0e318196-workload-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"
Oct 30 17:03:38.238156 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.238119    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07938ade-58fb-4ca9-a7a2-618a0e318196-credential-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"
Oct 30 17:03:38.238258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.238152    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07938ade-58fb-4ca9-a7a2-618a0e318196-istio-podinfo\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"
Oct 30 17:03:38.238318 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.238256    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q26g9\" (UniqueName: \"kubernetes.io/projected/07938ade-58fb-4ca9-a7a2-618a0e318196-kube-api-access-q26g9\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"
Oct 30 17:03:38.238373 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.238356    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07938ade-58fb-4ca9-a7a2-618a0e318196-istio-envoy\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"
Oct 30 17:03:38.238444 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.238372    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07938ade-58fb-4ca9-a7a2-618a0e318196-istio-data\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"
Oct 30 17:03:38.238444 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.238387    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07938ade-58fb-4ca9-a7a2-618a0e318196-istio-token\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"
Oct 30 17:03:38.238554 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.238457    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07938ade-58fb-4ca9-a7a2-618a0e318196-workload-certs\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"
Oct 30 17:03:38.238716 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.238674    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07938ade-58fb-4ca9-a7a2-618a0e318196-istiod-ca-cert\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"
Oct 30 17:03:38.238799 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.238730    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07938ade-58fb-4ca9-a7a2-618a0e318196-workload-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"
Oct 30 17:03:38.238799 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.238752    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07938ade-58fb-4ca9-a7a2-618a0e318196-credential-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"
Oct 30 17:03:38.240813 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.240780    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07938ade-58fb-4ca9-a7a2-618a0e318196-istio-envoy\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"
Oct 30 17:03:38.240924 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.240905    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07938ade-58fb-4ca9-a7a2-618a0e318196-istio-podinfo\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"
Oct 30 17:03:38.265467 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.265378    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q26g9\" (UniqueName: \"kubernetes.io/projected/07938ade-58fb-4ca9-a7a2-618a0e318196-kube-api-access-q26g9\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"
Oct 30 17:03:38.265467 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.265380    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07938ade-58fb-4ca9-a7a2-618a0e318196-istio-token\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"
Oct 30 17:03:38.332301 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.332268    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"
Oct 30 17:03:38.670464 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.670368    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"]
Oct 30 17:03:38.673999 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:03:38.673970    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07938ade_58fb_4ca9_a7a2_618a0e318196.slice/crio-d21abd95eb648d23e431633accec7d34d377fc2e3e8876f853e35cd02cf4f9f0 WatchSource:0}: Error finding container d21abd95eb648d23e431633accec7d34d377fc2e3e8876f853e35cd02cf4f9f0: Status 404 returned error can't find the container with id d21abd95eb648d23e431633accec7d34d377fc2e3e8876f853e35cd02cf4f9f0
Oct 30 17:03:38.676465 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.676431    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:03:38.676564 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.676498    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:03:38.676564 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:38.676550    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:03:39.232937 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:39.232895    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx" event={"ID":"07938ade-58fb-4ca9-a7a2-618a0e318196","Type":"ContainerStarted","Data":"1c5137a1215d1e467e13febb7f6a64b231936a9d442dfd8270240708acc723a8"}
Oct 30 17:03:39.233329 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:39.232962    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx" event={"ID":"07938ade-58fb-4ca9-a7a2-618a0e318196","Type":"ContainerStarted","Data":"d21abd95eb648d23e431633accec7d34d377fc2e3e8876f853e35cd02cf4f9f0"}
Oct 30 17:03:39.292053 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:39.291999    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx" podStartSLOduration=2.291982078 podStartE2EDuration="2.291982078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:03:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:03:39.288271193 +0000 UTC m=+995.723173400" watchObservedRunningTime="2026-10-30 17:03:39.291982078 +0000 UTC m=+995.726884285"
Oct 30 17:03:39.334925 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:39.334887    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"
Oct 30 17:03:39.335340 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:39.335308    4433 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 17:03:39.335617 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:39.335585    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx" podUID="07938ade-58fb-4ca9-a7a2-618a0e318196" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 17:03:40.332921 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:40.332882    4433 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 17:03:40.333304 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:40.332937    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx" podUID="07938ade-58fb-4ca9-a7a2-618a0e318196" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 17:03:41.333297 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:41.333255    4433 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 17:03:41.333830 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:41.333326    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx" podUID="07938ade-58fb-4ca9-a7a2-618a0e318196" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 17:03:42.334847 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:42.334807    4433 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 17:03:42.335728 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:42.335686    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx" podUID="07938ade-58fb-4ca9-a7a2-618a0e318196" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 17:03:43.332879 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:43.332835    4433 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 17:03:43.333073 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:43.332911    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx" podUID="07938ade-58fb-4ca9-a7a2-618a0e318196" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 17:03:44.332895 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:44.332855    4433 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 17:03:44.333425 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:44.332925    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx" podUID="07938ade-58fb-4ca9-a7a2-618a0e318196" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 17:03:45.333360 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:45.333319    4433 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 17:03:45.333798 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:45.333374    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx" podUID="07938ade-58fb-4ca9-a7a2-618a0e318196" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 17:03:45.453095 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:45.453062    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b718c2b-5zzxh"]
Oct 30 17:03:45.456362 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:45.456342    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b718c2b-5zzxh"
Oct 30 17:03:45.472571 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:45.472544    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:03:45.484554 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:45.484524    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\""
Oct 30 17:03:45.487743 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:45.487713    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b718c2b-5zzxh"]
Oct 30 17:03:45.596039 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:45.595943    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab1552df-b9c5-46fb-880f-ae189e69f391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b718c2b-5zzxh\" (UID: \"ab1552df-b9c5-46fb-880f-ae189e69f391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b718c2b-5zzxh"
Oct 30 17:03:45.596039 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:45.596018    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab1552df-b9c5-46fb-880f-ae189e69f391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b718c2b-5zzxh\" (UID: \"ab1552df-b9c5-46fb-880f-ae189e69f391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b718c2b-5zzxh"
Oct 30 17:03:45.596281 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:45.596059    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9bg8\" (UniqueName: \"kubernetes.io/projected/ab1552df-b9c5-46fb-880f-ae189e69f391-kube-api-access-p9bg8\") pod \"maas-api-revoke-sub-6b718c2b-5zzxh\" (UID: \"ab1552df-b9c5-46fb-880f-ae189e69f391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b718c2b-5zzxh"
Oct 30 17:03:45.697218 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:45.697179    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab1552df-b9c5-46fb-880f-ae189e69f391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b718c2b-5zzxh\" (UID: \"ab1552df-b9c5-46fb-880f-ae189e69f391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b718c2b-5zzxh"
Oct 30 17:03:45.697434 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:45.697245    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab1552df-b9c5-46fb-880f-ae189e69f391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b718c2b-5zzxh\" (UID: \"ab1552df-b9c5-46fb-880f-ae189e69f391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b718c2b-5zzxh"
Oct 30 17:03:45.697434 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:45.697263    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9bg8\" (UniqueName: \"kubernetes.io/projected/ab1552df-b9c5-46fb-880f-ae189e69f391-kube-api-access-p9bg8\") pod \"maas-api-revoke-sub-6b718c2b-5zzxh\" (UID: \"ab1552df-b9c5-46fb-880f-ae189e69f391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b718c2b-5zzxh"
Oct 30 17:03:45.697917 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:45.697891    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab1552df-b9c5-46fb-880f-ae189e69f391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b718c2b-5zzxh\" (UID: \"ab1552df-b9c5-46fb-880f-ae189e69f391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b718c2b-5zzxh"
Oct 30 17:03:45.699746 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:45.699719    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab1552df-b9c5-46fb-880f-ae189e69f391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b718c2b-5zzxh\" (UID: \"ab1552df-b9c5-46fb-880f-ae189e69f391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b718c2b-5zzxh"
Oct 30 17:03:45.720493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:45.720458    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9bg8\" (UniqueName: \"kubernetes.io/projected/ab1552df-b9c5-46fb-880f-ae189e69f391-kube-api-access-p9bg8\") pod \"maas-api-revoke-sub-6b718c2b-5zzxh\" (UID: \"ab1552df-b9c5-46fb-880f-ae189e69f391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b718c2b-5zzxh"
Oct 30 17:03:45.769988 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:45.769952    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b718c2b-5zzxh"
Oct 30 17:03:46.082220 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:46.082192    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b718c2b-5zzxh"]
Oct 30 17:03:46.084866 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:03:46.084831    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab1552df_b9c5_46fb_880f_ae189e69f391.slice/crio-d7e845d0b6300d746700ab5b5245c4cda0b0979ac7c6164003deb2be6ee0e500 WatchSource:0}: Error finding container d7e845d0b6300d746700ab5b5245c4cda0b0979ac7c6164003deb2be6ee0e500: Status 404 returned error can't find the container with id d7e845d0b6300d746700ab5b5245c4cda0b0979ac7c6164003deb2be6ee0e500
Oct 30 17:03:46.304946 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:46.304899    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b718c2b-5zzxh" event={"ID":"ab1552df-b9c5-46fb-880f-ae189e69f391","Type":"ContainerStarted","Data":"d7e845d0b6300d746700ab5b5245c4cda0b0979ac7c6164003deb2be6ee0e500"}
Oct 30 17:03:46.349356 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:46.349254    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"
Oct 30 17:03:46.350172 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:46.350143    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"
Oct 30 17:03:46.351058 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:46.351037    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"
Oct 30 17:03:47.312906 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:47.312802    4433 generic.go:356] "Generic (PLEG): container finished" podID="ab1552df-b9c5-46fb-880f-ae189e69f391" containerID="56414be6672f899945b39cf7781305f5eb09601e16bac907560a6d199bb387d1" exitCode=0
Oct 30 17:03:47.313055 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:47.312894    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b718c2b-5zzxh" event={"ID":"ab1552df-b9c5-46fb-880f-ae189e69f391","Type":"ContainerDied","Data":"56414be6672f899945b39cf7781305f5eb09601e16bac907560a6d199bb387d1"}
Oct 30 17:03:48.473856 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:48.473829    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b718c2b-5zzxh"
Oct 30 17:03:48.638357 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:48.638267    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab1552df-b9c5-46fb-880f-ae189e69f391-kube-api-access-p9bg8\" (UniqueName: \"kubernetes.io/projected/ab1552df-b9c5-46fb-880f-ae189e69f391-kube-api-access-p9bg8\") pod \"ab1552df-b9c5-46fb-880f-ae189e69f391\" (UID: \"ab1552df-b9c5-46fb-880f-ae189e69f391\") "
Oct 30 17:03:48.638357 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:48.638323    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab1552df-b9c5-46fb-880f-ae189e69f391-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab1552df-b9c5-46fb-880f-ae189e69f391-maas-api-xks-ca\") pod \"ab1552df-b9c5-46fb-880f-ae189e69f391\" (UID: \"ab1552df-b9c5-46fb-880f-ae189e69f391\") "
Oct 30 17:03:48.638646 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:48.638373    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab1552df-b9c5-46fb-880f-ae189e69f391-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab1552df-b9c5-46fb-880f-ae189e69f391-maas-api-openshift-ca\") pod \"ab1552df-b9c5-46fb-880f-ae189e69f391\" (UID: \"ab1552df-b9c5-46fb-880f-ae189e69f391\") "
Oct 30 17:03:48.638869 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:48.638841    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab1552df-b9c5-46fb-880f-ae189e69f391-maas-api-openshift-ca" pod "ab1552df-b9c5-46fb-880f-ae189e69f391" (UID: "ab1552df-b9c5-46fb-880f-ae189e69f391"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:03:48.639014 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:48.638998    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab1552df-b9c5-46fb-880f-ae189e69f391-maas-api-openshift-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:03:48.640493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:48.640470    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab1552df-b9c5-46fb-880f-ae189e69f391-maas-api-xks-ca" pod "ab1552df-b9c5-46fb-880f-ae189e69f391" (UID: "ab1552df-b9c5-46fb-880f-ae189e69f391"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:03:48.640554 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:48.640499    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab1552df-b9c5-46fb-880f-ae189e69f391-kube-api-access-p9bg8" pod "ab1552df-b9c5-46fb-880f-ae189e69f391" (UID: "ab1552df-b9c5-46fb-880f-ae189e69f391"). InnerVolumeSpecName "kube-api-access-p9bg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:03:48.740113 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:48.740074    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9bg8\" (UniqueName: \"kubernetes.io/projected/ab1552df-b9c5-46fb-880f-ae189e69f391-kube-api-access-p9bg8\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:03:48.740113 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:48.740107    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab1552df-b9c5-46fb-880f-ae189e69f391-maas-api-xks-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:03:49.327520 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:49.327490    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b718c2b-5zzxh"
Oct 30 17:03:49.327690 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:49.327525    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b718c2b-5zzxh" event={"ID":"ab1552df-b9c5-46fb-880f-ae189e69f391","Type":"ContainerDied","Data":"d7e845d0b6300d746700ab5b5245c4cda0b0979ac7c6164003deb2be6ee0e500"}
Oct 30 17:03:49.327690 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:49.327602    4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7e845d0b6300d746700ab5b5245c4cda0b0979ac7c6164003deb2be6ee0e500"
Oct 30 17:03:52.382297 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:52.382261    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"]
Oct 30 17:03:52.382855 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:52.382841    4433 cpu_manager.go:414] "containerMap: removing container" podUID="ab1552df-b9c5-46fb-880f-ae189e69f391" containerName="revoke-keys"
Oct 30 17:03:52.382899 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:52.382856    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab1552df-b9c5-46fb-880f-ae189e69f391" containerName="revoke-keys"
Oct 30 17:03:52.382966 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:52.382956    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab1552df-b9c5-46fb-880f-ae189e69f391" containerName="revoke-keys"
Oct 30 17:03:52.392044 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:52.392017    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"
Oct 30 17:03:52.425588 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:52.425550    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"]
Oct 30 17:03:52.462861 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:52.462822    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:03:52.491074 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:52.491031    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6431be3-b51d-40bc-bc87-ddf189905e14-istio-data\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"
Oct 30 17:03:52.491257 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:52.491081    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvcc7\" (UniqueName: \"kubernetes.io/projected/e6431be3-b51d-40bc-bc87-ddf189905e14-kube-api-access-fvcc7\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"
Oct 30 17:03:52.491257 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:52.491186    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6431be3-b51d-40bc-bc87-ddf189905e14-istio-podinfo\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"
Oct 30 17:03:52.491257 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:52.491236    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6431be3-b51d-40bc-bc87-ddf189905e14-workload-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"
Oct 30 17:03:52.491384 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:52.491301    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6431be3-b51d-40bc-bc87-ddf189905e14-istio-envoy\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"
Oct 30 17:03:52.491384 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:52.491329    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6431be3-b51d-40bc-bc87-ddf189905e14-istiod-ca-cert\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"
Oct 30 17:03:52.491536 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:52.491478    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6431be3-b51d-40bc-bc87-ddf189905e14-workload-certs\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"
Oct 30 17:03:52.491625 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:52.491609    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6431be3-b51d-40bc-bc87-ddf189905e14-credential-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"
Oct 30 17:03:52.491677 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:52.491639    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6431be3-b51d-40bc-bc87-ddf189905e14-istio-token\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"
Oct 30 17:03:52.592783 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:52.592739    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6431be3-b51d-40bc-bc87-ddf189905e14-istio-data\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"
Oct 30 17:03:52.592957 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:52.592795    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvcc7\" (UniqueName: \"kubernetes.io/projected/e6431be3-b51d-40bc-bc87-ddf189905e14-kube-api-access-fvcc7\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"
Oct 30 17:03:52.592957 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:52.592835    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6431be3-b51d-40bc-bc87-ddf189905e14-istio-podinfo\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"
Oct 30 17:03:52.592957 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:52.592880    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6431be3-b51d-40bc-bc87-ddf189905e14-workload-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"
Oct 30 17:03:52.592957 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:52.592926    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6431be3-b51d-40bc-bc87-ddf189905e14-istio-envoy\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"
Oct 30 17:03:52.592957 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:52.592947    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6431be3-b51d-40bc-bc87-ddf189905e14-istiod-ca-cert\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"
Oct 30 17:03:52.593280 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:52.593038    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6431be3-b51d-40bc-bc87-ddf189905e14-workload-certs\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"
Oct 30 17:03:52.593280 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:52.593169    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6431be3-b51d-40bc-bc87-ddf189905e14-credential-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"
Oct 30 17:03:52.593280 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:52.593172    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6431be3-b51d-40bc-bc87-ddf189905e14-istio-data\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"
Oct 30 17:03:52.593280 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:52.593209    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6431be3-b51d-40bc-bc87-ddf189905e14-istio-token\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"
Oct 30 17:03:52.593280 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:52.593248    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6431be3-b51d-40bc-bc87-ddf189905e14-workload-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"
Oct 30 17:03:52.593750 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:52.593709    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6431be3-b51d-40bc-bc87-ddf189905e14-credential-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"
Oct 30 17:03:52.593901 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:52.593785    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6431be3-b51d-40bc-bc87-ddf189905e14-workload-certs\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"
Oct 30 17:03:52.593901 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:52.593877    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6431be3-b51d-40bc-bc87-ddf189905e14-istiod-ca-cert\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"
Oct 30 17:03:52.595571 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:52.595542    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6431be3-b51d-40bc-bc87-ddf189905e14-istio-envoy\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"
Oct 30 17:03:52.595804 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:52.595783    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6431be3-b51d-40bc-bc87-ddf189905e14-istio-podinfo\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"
Oct 30 17:03:52.615799 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:52.615765    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvcc7\" (UniqueName: \"kubernetes.io/projected/e6431be3-b51d-40bc-bc87-ddf189905e14-kube-api-access-fvcc7\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"
Oct 30 17:03:52.615939 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:52.615879    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6431be3-b51d-40bc-bc87-ddf189905e14-istio-token\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"
Oct 30 17:03:52.705497 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:52.705463    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"
Oct 30 17:03:53.038341 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:53.038311    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"]
Oct 30 17:03:53.041263 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:03:53.041234    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6431be3_b51d_40bc_bc87_ddf189905e14.slice/crio-e3c513ec4ce23db0c68627aa3b4f298c145c083e01887ba76ec9f436aa7372a2 WatchSource:0}: Error finding container e3c513ec4ce23db0c68627aa3b4f298c145c083e01887ba76ec9f436aa7372a2: Status 404 returned error can't find the container with id e3c513ec4ce23db0c68627aa3b4f298c145c083e01887ba76ec9f436aa7372a2
Oct 30 17:03:53.044666 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:53.044606    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:03:53.044949 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:53.044921    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:03:53.045152 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:53.045132    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:03:53.361735 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:53.359032    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz" event={"ID":"e6431be3-b51d-40bc-bc87-ddf189905e14","Type":"ContainerStarted","Data":"f15641e0650eb18d0e9ac76f8e575f111731df480cebe92de1b855a61cab3a82"}
Oct 30 17:03:53.361735 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:53.359146    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz" event={"ID":"e6431be3-b51d-40bc-bc87-ddf189905e14","Type":"ContainerStarted","Data":"e3c513ec4ce23db0c68627aa3b4f298c145c083e01887ba76ec9f436aa7372a2"}
Oct 30 17:03:53.409755 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:53.409694    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz" podStartSLOduration=1.409673894 podStartE2EDuration="1.409673894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:03:53.404997747 +0000 UTC m=+1009.839899956" watchObservedRunningTime="2026-10-30 17:03:53.409673894 +0000 UTC m=+1009.844576105"
Oct 30 17:03:53.705598 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:53.705566    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"
Oct 30 17:03:54.714300 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:54.714267    4433 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 17:03:54.714896 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:54.714869    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz" podUID="e6431be3-b51d-40bc-bc87-ddf189905e14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 17:03:55.036490 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:55.032558    4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" podUID="458eba2e-eb03-452a-bcf4-a5e91da83acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:03:55.100549 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:55.100495    4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:03:55.224555 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:55.224509    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:03:55.356638 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:55.356531    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:03:55.363538 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:55.363491    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" podUID="458eba2e-eb03-452a-bcf4-a5e91da83acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:03:55.411541 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:55.411492    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 17:03:55.445430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:55.443946    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:03:55.463429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:55.458558    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:03:55.512539 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:55.512491    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:03:55.711426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:55.711247    4433 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 17:03:55.711426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:55.711306    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz" podUID="e6431be3-b51d-40bc-bc87-ddf189905e14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 17:03:56.261063 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:56.252584    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm" podUID="bf43bf3d-f126-40a1-a24f-10214a85ceb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:03:56.420429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:56.409513    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:03:56.420429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:56.410340    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:03:56.437682 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:56.422444    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" podUID="458eba2e-eb03-452a-bcf4-a5e91da83acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 17:03:56.437682 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:56.422852    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:03:56.437682 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:56.424145    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:03:56.437682 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:56.424491    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 17:03:56.454798 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:56.446593    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:03:56.454798 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:56.447020    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 17:03:56.524492 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:56.511521    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:03:56.547434 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:56.539491    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:03:56.547434 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:56.539967    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:03:56.724000 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:56.723965    4433 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 17:03:56.724274 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:56.724242    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz" podUID="e6431be3-b51d-40bc-bc87-ddf189905e14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 17:03:57.156429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:57.145923    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f603c30-ldbbj"]
Oct 30 17:03:57.160535 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:57.160507    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f603c30-ldbbj"
Oct 30 17:03:57.195071 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:57.194479    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\""
Oct 30 17:03:57.195071 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:57.194550    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:03:57.237173 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:57.236703    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f603c30-ldbbj"]
Oct 30 17:03:57.280466 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:57.272368    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x6kf\" (UniqueName: \"kubernetes.io/projected/9587981e-f995-4628-8131-50fcbda9aaf3-kube-api-access-5x6kf\") pod \"maas-api-revoke-sub-2f603c30-ldbbj\" (UID: \"9587981e-f995-4628-8131-50fcbda9aaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f603c30-ldbbj"
Oct 30 17:03:57.280466 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:57.272475    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9587981e-f995-4628-8131-50fcbda9aaf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f603c30-ldbbj\" (UID: \"9587981e-f995-4628-8131-50fcbda9aaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f603c30-ldbbj"
Oct 30 17:03:57.280466 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:57.272723    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9587981e-f995-4628-8131-50fcbda9aaf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f603c30-ldbbj\" (UID: \"9587981e-f995-4628-8131-50fcbda9aaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f603c30-ldbbj"
Oct 30 17:03:57.329221 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:57.293532    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:03:57.329221 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:57.294126    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 17:03:57.486885 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:57.486136    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" podUID="458eba2e-eb03-452a-bcf4-a5e91da83acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 17:03:57.487737 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:57.487146    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 17:03:57.595774 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:03:57.582803    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:03:57.656088 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:57.656021    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 17:03:57.696931 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:57.696875    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 17:03:57.762486 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:57.716664    4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:03:57.776033 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:57.775891    4433 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 17:03:57.778445 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:57.776329    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz" podUID="e6431be3-b51d-40bc-bc87-ddf189905e14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 17:03:57.778770 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:57.778739    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:03:57.920045 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:57.919878    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 30 17:03:57.948067 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:57.946748    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9587981e-f995-4628-8131-50fcbda9aaf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f603c30-ldbbj\" (UID: \"9587981e-f995-4628-8131-50fcbda9aaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f603c30-ldbbj"
Oct 30 17:03:57.948067 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:57.947084    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9587981e-f995-4628-8131-50fcbda9aaf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f603c30-ldbbj\" (UID: \"9587981e-f995-4628-8131-50fcbda9aaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f603c30-ldbbj"
Oct 30 17:03:57.948067 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:57.947201    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x6kf\" (UniqueName: \"kubernetes.io/projected/9587981e-f995-4628-8131-50fcbda9aaf3-kube-api-access-5x6kf\") pod \"maas-api-revoke-sub-2f603c30-ldbbj\" (UID: \"9587981e-f995-4628-8131-50fcbda9aaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f603c30-ldbbj"
Oct 30 17:03:57.948067 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:57.947920    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9587981e-f995-4628-8131-50fcbda9aaf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f603c30-ldbbj\" (UID: \"9587981e-f995-4628-8131-50fcbda9aaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f603c30-ldbbj"
Oct 30 17:03:57.986429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:57.979165    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 17:03:57.986429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:57.979226    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:03:57.986429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:57.979260    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:03:57.986429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:57.979292    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 17:03:57.986429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:57.979325    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 17:03:57.989432 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:57.988879    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:03:57.996432 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:57.996010    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9587981e-f995-4628-8131-50fcbda9aaf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f603c30-ldbbj\" (UID: \"9587981e-f995-4628-8131-50fcbda9aaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f603c30-ldbbj"
Oct 30 17:03:57.997314 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:57.996687    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x6kf\" (UniqueName: \"kubernetes.io/projected/9587981e-f995-4628-8131-50fcbda9aaf3-kube-api-access-5x6kf\") pod \"maas-api-revoke-sub-2f603c30-ldbbj\" (UID: \"9587981e-f995-4628-8131-50fcbda9aaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f603c30-ldbbj"
Oct 30 17:03:58.106725 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:58.106645    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f603c30-ldbbj"
Oct 30 17:03:58.477497 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:58.477460    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f603c30-ldbbj"]
Oct 30 17:03:58.481924 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:03:58.481895    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9587981e_f995_4628_8131_50fcbda9aaf3.slice/crio-46439043d0069719cb07bda7e336d51b412bc39a5102b731440c536ca4df6f5e WatchSource:0}: Error finding container 46439043d0069719cb07bda7e336d51b412bc39a5102b731440c536ca4df6f5e: Status 404 returned error can't find the container with id 46439043d0069719cb07bda7e336d51b412bc39a5102b731440c536ca4df6f5e
Oct 30 17:03:58.705974 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:58.705940    4433 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 17:03:58.706104 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:58.705991    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz" podUID="e6431be3-b51d-40bc-bc87-ddf189905e14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 17:03:58.946009 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:58.945976    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f603c30-ldbbj" event={"ID":"9587981e-f995-4628-8131-50fcbda9aaf3","Type":"ContainerStarted","Data":"aefbee01a0d85052512d2db507a4de61e8060b1a7fdcd02d26c8b4be39178f24"}
Oct 30 17:03:58.946189 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:58.946037    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f603c30-ldbbj" event={"ID":"9587981e-f995-4628-8131-50fcbda9aaf3","Type":"ContainerStarted","Data":"46439043d0069719cb07bda7e336d51b412bc39a5102b731440c536ca4df6f5e"}
Oct 30 17:03:59.004603 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:59.004548    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f603c30-ldbbj" podStartSLOduration=3.004533123 podStartE2EDuration="3.004533123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:03:59.001169056 +0000 UTC m=+1015.436071283" watchObservedRunningTime="2026-10-30 17:03:59.004533123 +0000 UTC m=+1015.439435328"
Oct 30 17:03:59.706668 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:59.706628    4433 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 17:03:59.707151 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:59.706682    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz" podUID="e6431be3-b51d-40bc-bc87-ddf189905e14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 17:03:59.975440 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:59.974241    4433 generic.go:356] "Generic (PLEG): container finished" podID="9587981e-f995-4628-8131-50fcbda9aaf3" containerID="aefbee01a0d85052512d2db507a4de61e8060b1a7fdcd02d26c8b4be39178f24" exitCode=0
Oct 30 17:03:59.975440 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:03:59.974444    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f603c30-ldbbj" event={"ID":"9587981e-f995-4628-8131-50fcbda9aaf3","Type":"ContainerDied","Data":"aefbee01a0d85052512d2db507a4de61e8060b1a7fdcd02d26c8b4be39178f24"}
Oct 30 17:04:00.705944 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:00.705912    4433 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 17:04:00.706131 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:00.705977    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz" podUID="e6431be3-b51d-40bc-bc87-ddf189905e14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 17:04:01.168689 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:01.168664    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f603c30-ldbbj"
Oct 30 17:04:01.226116 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:01.226085    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9587981e-f995-4628-8131-50fcbda9aaf3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9587981e-f995-4628-8131-50fcbda9aaf3-maas-api-xks-ca\") pod \"9587981e-f995-4628-8131-50fcbda9aaf3\" (UID: \"9587981e-f995-4628-8131-50fcbda9aaf3\") "
Oct 30 17:04:01.226314 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:01.226295    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9587981e-f995-4628-8131-50fcbda9aaf3-kube-api-access-5x6kf\" (UniqueName: \"kubernetes.io/projected/9587981e-f995-4628-8131-50fcbda9aaf3-kube-api-access-5x6kf\") pod \"9587981e-f995-4628-8131-50fcbda9aaf3\" (UID: \"9587981e-f995-4628-8131-50fcbda9aaf3\") "
Oct 30 17:04:01.226503 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:01.226488    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9587981e-f995-4628-8131-50fcbda9aaf3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9587981e-f995-4628-8131-50fcbda9aaf3-maas-api-openshift-ca\") pod \"9587981e-f995-4628-8131-50fcbda9aaf3\" (UID: \"9587981e-f995-4628-8131-50fcbda9aaf3\") "
Oct 30 17:04:01.227017 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:01.226966    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9587981e-f995-4628-8131-50fcbda9aaf3-maas-api-openshift-ca" pod "9587981e-f995-4628-8131-50fcbda9aaf3" (UID: "9587981e-f995-4628-8131-50fcbda9aaf3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:04:01.227166 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:01.227105    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9587981e-f995-4628-8131-50fcbda9aaf3-maas-api-openshift-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:04:01.228630 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:01.228602    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9587981e-f995-4628-8131-50fcbda9aaf3-kube-api-access-5x6kf" pod "9587981e-f995-4628-8131-50fcbda9aaf3" (UID: "9587981e-f995-4628-8131-50fcbda9aaf3"). InnerVolumeSpecName "kube-api-access-5x6kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:04:01.229007 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:01.228969    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9587981e-f995-4628-8131-50fcbda9aaf3-maas-api-xks-ca" pod "9587981e-f995-4628-8131-50fcbda9aaf3" (UID: "9587981e-f995-4628-8131-50fcbda9aaf3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:04:01.328888 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:01.328852    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9587981e-f995-4628-8131-50fcbda9aaf3-maas-api-xks-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:04:01.328888 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:01.328883    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x6kf\" (UniqueName: \"kubernetes.io/projected/9587981e-f995-4628-8131-50fcbda9aaf3-kube-api-access-5x6kf\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:04:01.706072 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:01.706043    4433 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 17:04:01.706212 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:01.706109    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz" podUID="e6431be3-b51d-40bc-bc87-ddf189905e14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 17:04:01.988681 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:01.988581    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f603c30-ldbbj"
Oct 30 17:04:01.988681 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:01.988594    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f603c30-ldbbj" event={"ID":"9587981e-f995-4628-8131-50fcbda9aaf3","Type":"ContainerDied","Data":"46439043d0069719cb07bda7e336d51b412bc39a5102b731440c536ca4df6f5e"}
Oct 30 17:04:01.988681 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:01.988653    4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46439043d0069719cb07bda7e336d51b412bc39a5102b731440c536ca4df6f5e"
Oct 30 17:04:03.631550 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:03.631515    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"
Oct 30 17:04:03.632443 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:03.632419    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"
Oct 30 17:04:03.633500 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:03.633477    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"
Oct 30 17:04:04.445284 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:04.445244    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c5e907b-8p2sm"]
Oct 30 17:04:04.445886 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:04.445870    4433 cpu_manager.go:414] "containerMap: removing container" podUID="9587981e-f995-4628-8131-50fcbda9aaf3" containerName="revoke-keys"
Oct 30 17:04:04.445940 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:04.445888    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9587981e-f995-4628-8131-50fcbda9aaf3" containerName="revoke-keys"
Oct 30 17:04:04.445993 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:04.445985    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="9587981e-f995-4628-8131-50fcbda9aaf3" containerName="revoke-keys"
Oct 30 17:04:04.449349 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:04.449330    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5e907b-8p2sm"
Oct 30 17:04:04.476586 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:04.476555    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:04:04.476817 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:04.476574    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\""
Oct 30 17:04:04.481350 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:04.481323    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c5e907b-8p2sm"]
Oct 30 17:04:04.576738 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:04.576623    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs757\" (UniqueName: \"kubernetes.io/projected/548f3d48-17cc-4345-82e2-2a1babdeccc5-kube-api-access-bs757\") pod \"maas-api-revoke-sub-0c5e907b-8p2sm\" (UID: \"548f3d48-17cc-4345-82e2-2a1babdeccc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5e907b-8p2sm"
Oct 30 17:04:04.576931 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:04.576811    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/548f3d48-17cc-4345-82e2-2a1babdeccc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c5e907b-8p2sm\" (UID: \"548f3d48-17cc-4345-82e2-2a1babdeccc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5e907b-8p2sm"
Oct 30 17:04:04.577148 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:04.576960    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/548f3d48-17cc-4345-82e2-2a1babdeccc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c5e907b-8p2sm\" (UID: \"548f3d48-17cc-4345-82e2-2a1babdeccc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5e907b-8p2sm"
Oct 30 17:04:04.678832 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:04.678797    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/548f3d48-17cc-4345-82e2-2a1babdeccc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c5e907b-8p2sm\" (UID: \"548f3d48-17cc-4345-82e2-2a1babdeccc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5e907b-8p2sm"
Oct 30 17:04:04.679212 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:04.678920    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/548f3d48-17cc-4345-82e2-2a1babdeccc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c5e907b-8p2sm\" (UID: \"548f3d48-17cc-4345-82e2-2a1babdeccc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5e907b-8p2sm"
Oct 30 17:04:04.679212 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:04.679023    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs757\" (UniqueName: \"kubernetes.io/projected/548f3d48-17cc-4345-82e2-2a1babdeccc5-kube-api-access-bs757\") pod \"maas-api-revoke-sub-0c5e907b-8p2sm\" (UID: \"548f3d48-17cc-4345-82e2-2a1babdeccc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5e907b-8p2sm"
Oct 30 17:04:04.679571 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:04.679539    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/548f3d48-17cc-4345-82e2-2a1babdeccc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c5e907b-8p2sm\" (UID: \"548f3d48-17cc-4345-82e2-2a1babdeccc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5e907b-8p2sm"
Oct 30 17:04:04.681157 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:04.681138    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/548f3d48-17cc-4345-82e2-2a1babdeccc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c5e907b-8p2sm\" (UID: \"548f3d48-17cc-4345-82e2-2a1babdeccc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5e907b-8p2sm"
Oct 30 17:04:04.701774 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:04.701704    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs757\" (UniqueName: \"kubernetes.io/projected/548f3d48-17cc-4345-82e2-2a1babdeccc5-kube-api-access-bs757\") pod \"maas-api-revoke-sub-0c5e907b-8p2sm\" (UID: \"548f3d48-17cc-4345-82e2-2a1babdeccc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5e907b-8p2sm"
Oct 30 17:04:04.763029 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:04.762997    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5e907b-8p2sm"
Oct 30 17:04:05.080831 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:05.080806    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c5e907b-8p2sm"]
Oct 30 17:04:05.083514 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:04:05.083474    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod548f3d48_17cc_4345_82e2_2a1babdeccc5.slice/crio-682601c6d9c02ca0f044c0af5278a3c812a209e89527c8df0a26736a2dfe2730 WatchSource:0}: Error finding container 682601c6d9c02ca0f044c0af5278a3c812a209e89527c8df0a26736a2dfe2730: Status 404 returned error can't find the container with id 682601c6d9c02ca0f044c0af5278a3c812a209e89527c8df0a26736a2dfe2730
Oct 30 17:04:06.018737 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:06.018696    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5e907b-8p2sm" event={"ID":"548f3d48-17cc-4345-82e2-2a1babdeccc5","Type":"ContainerStarted","Data":"2f3fc3ac65342a91e0bae958a1965c53812ec254dc04e3844acda8351308af58"}
Oct 30 17:04:06.019139 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:06.018762    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5e907b-8p2sm" event={"ID":"548f3d48-17cc-4345-82e2-2a1babdeccc5","Type":"ContainerStarted","Data":"682601c6d9c02ca0f044c0af5278a3c812a209e89527c8df0a26736a2dfe2730"}
Oct 30 17:04:06.063969 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:06.063915    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5e907b-8p2sm" podStartSLOduration=2.063897543 podStartE2EDuration="2.063897543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:04:06.059915762 +0000 UTC m=+1022.494817969" watchObservedRunningTime="2026-10-30 17:04:06.063897543 +0000 UTC m=+1022.498799777"
Oct 30 17:04:07.026534 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:07.026490    4433 generic.go:356] "Generic (PLEG): container finished" podID="548f3d48-17cc-4345-82e2-2a1babdeccc5" containerID="2f3fc3ac65342a91e0bae958a1965c53812ec254dc04e3844acda8351308af58" exitCode=0
Oct 30 17:04:07.026941 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:07.026573    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5e907b-8p2sm" event={"ID":"548f3d48-17cc-4345-82e2-2a1babdeccc5","Type":"ContainerDied","Data":"2f3fc3ac65342a91e0bae958a1965c53812ec254dc04e3844acda8351308af58"}
Oct 30 17:04:08.183428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:08.183392    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5e907b-8p2sm"
Oct 30 17:04:08.328237 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:08.328146    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/548f3d48-17cc-4345-82e2-2a1babdeccc5-kube-api-access-bs757\" (UniqueName: \"kubernetes.io/projected/548f3d48-17cc-4345-82e2-2a1babdeccc5-kube-api-access-bs757\") pod \"548f3d48-17cc-4345-82e2-2a1babdeccc5\" (UID: \"548f3d48-17cc-4345-82e2-2a1babdeccc5\") "
Oct 30 17:04:08.328237 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:08.328196    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/548f3d48-17cc-4345-82e2-2a1babdeccc5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/548f3d48-17cc-4345-82e2-2a1babdeccc5-maas-api-openshift-ca\") pod \"548f3d48-17cc-4345-82e2-2a1babdeccc5\" (UID: \"548f3d48-17cc-4345-82e2-2a1babdeccc5\") "
Oct 30 17:04:08.328237 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:08.328224    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/548f3d48-17cc-4345-82e2-2a1babdeccc5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/548f3d48-17cc-4345-82e2-2a1babdeccc5-maas-api-xks-ca\") pod \"548f3d48-17cc-4345-82e2-2a1babdeccc5\" (UID: \"548f3d48-17cc-4345-82e2-2a1babdeccc5\") "
Oct 30 17:04:08.328671 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:08.328644    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/548f3d48-17cc-4345-82e2-2a1babdeccc5-maas-api-openshift-ca" pod "548f3d48-17cc-4345-82e2-2a1babdeccc5" (UID: "548f3d48-17cc-4345-82e2-2a1babdeccc5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:04:08.329031 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:08.329012    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/548f3d48-17cc-4345-82e2-2a1babdeccc5-maas-api-openshift-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:04:08.330382 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:08.330359    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/548f3d48-17cc-4345-82e2-2a1babdeccc5-maas-api-xks-ca" pod "548f3d48-17cc-4345-82e2-2a1babdeccc5" (UID: "548f3d48-17cc-4345-82e2-2a1babdeccc5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:04:08.330513 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:08.330489    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/548f3d48-17cc-4345-82e2-2a1babdeccc5-kube-api-access-bs757" pod "548f3d48-17cc-4345-82e2-2a1babdeccc5" (UID: "548f3d48-17cc-4345-82e2-2a1babdeccc5"). InnerVolumeSpecName "kube-api-access-bs757". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:04:08.430460 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:08.430394    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs757\" (UniqueName: \"kubernetes.io/projected/548f3d48-17cc-4345-82e2-2a1babdeccc5-kube-api-access-bs757\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:04:08.430460 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:08.430457    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/548f3d48-17cc-4345-82e2-2a1babdeccc5-maas-api-xks-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:04:09.040940 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:09.040909    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5e907b-8p2sm"
Oct 30 17:04:09.041100 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:09.040939    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5e907b-8p2sm" event={"ID":"548f3d48-17cc-4345-82e2-2a1babdeccc5","Type":"ContainerDied","Data":"682601c6d9c02ca0f044c0af5278a3c812a209e89527c8df0a26736a2dfe2730"}
Oct 30 17:04:09.041100 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:09.040990    4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="682601c6d9c02ca0f044c0af5278a3c812a209e89527c8df0a26736a2dfe2730"
Oct 30 17:04:17.133084 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:17.133004    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-470be499-sw9hh"]
Oct 30 17:04:17.133583 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:17.133568    4433 cpu_manager.go:414] "containerMap: removing container" podUID="548f3d48-17cc-4345-82e2-2a1babdeccc5" containerName="revoke-keys"
Oct 30 17:04:17.133637 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:17.133584    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="548f3d48-17cc-4345-82e2-2a1babdeccc5" containerName="revoke-keys"
Oct 30 17:04:17.133697 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:17.133687    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="548f3d48-17cc-4345-82e2-2a1babdeccc5" containerName="revoke-keys"
Oct 30 17:04:17.137506 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:17.137487    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-470be499-sw9hh"
Oct 30 17:04:17.152098 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:17.152073    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hsc5\" (UniqueName: \"kubernetes.io/projected/aaf2e083-ca03-4482-8bc1-97d50fc52dec-kube-api-access-5hsc5\") pod \"maas-api-revoke-sub-470be499-sw9hh\" (UID: \"aaf2e083-ca03-4482-8bc1-97d50fc52dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470be499-sw9hh"
Oct 30 17:04:17.152228 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:17.152211    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaf2e083-ca03-4482-8bc1-97d50fc52dec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-470be499-sw9hh\" (UID: \"aaf2e083-ca03-4482-8bc1-97d50fc52dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470be499-sw9hh"
Oct 30 17:04:17.152286 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:17.152274    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaf2e083-ca03-4482-8bc1-97d50fc52dec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-470be499-sw9hh\" (UID: \"aaf2e083-ca03-4482-8bc1-97d50fc52dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470be499-sw9hh"
Oct 30 17:04:17.153158 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:17.153136    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:04:17.167592 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:17.167556    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\""
Oct 30 17:04:17.168779 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:17.168754    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-470be499-sw9hh"]
Oct 30 17:04:17.254063 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:17.254025    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaf2e083-ca03-4482-8bc1-97d50fc52dec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-470be499-sw9hh\" (UID: \"aaf2e083-ca03-4482-8bc1-97d50fc52dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470be499-sw9hh"
Oct 30 17:04:17.254254 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:17.254179    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaf2e083-ca03-4482-8bc1-97d50fc52dec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-470be499-sw9hh\" (UID: \"aaf2e083-ca03-4482-8bc1-97d50fc52dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470be499-sw9hh"
Oct 30 17:04:17.254345 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:17.254329    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hsc5\" (UniqueName: \"kubernetes.io/projected/aaf2e083-ca03-4482-8bc1-97d50fc52dec-kube-api-access-5hsc5\") pod \"maas-api-revoke-sub-470be499-sw9hh\" (UID: \"aaf2e083-ca03-4482-8bc1-97d50fc52dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470be499-sw9hh"
Oct 30 17:04:17.254828 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:17.254809    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaf2e083-ca03-4482-8bc1-97d50fc52dec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-470be499-sw9hh\" (UID: \"aaf2e083-ca03-4482-8bc1-97d50fc52dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470be499-sw9hh"
Oct 30 17:04:17.256343 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:17.256324    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaf2e083-ca03-4482-8bc1-97d50fc52dec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-470be499-sw9hh\" (UID: \"aaf2e083-ca03-4482-8bc1-97d50fc52dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470be499-sw9hh"
Oct 30 17:04:17.277861 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:17.277830    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hsc5\" (UniqueName: \"kubernetes.io/projected/aaf2e083-ca03-4482-8bc1-97d50fc52dec-kube-api-access-5hsc5\") pod \"maas-api-revoke-sub-470be499-sw9hh\" (UID: \"aaf2e083-ca03-4482-8bc1-97d50fc52dec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-470be499-sw9hh"
Oct 30 17:04:17.451305 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:17.451271    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-470be499-sw9hh"
Oct 30 17:04:17.770138 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:17.770103    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-470be499-sw9hh"]
Oct 30 17:04:17.771052 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:04:17.771026    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaf2e083_ca03_4482_8bc1_97d50fc52dec.slice/crio-e7f4710eee7e15d1eca4fe29f6421607f052c630ba370d5f4e6948b24cf60f65 WatchSource:0}: Error finding container e7f4710eee7e15d1eca4fe29f6421607f052c630ba370d5f4e6948b24cf60f65: Status 404 returned error can't find the container with id e7f4710eee7e15d1eca4fe29f6421607f052c630ba370d5f4e6948b24cf60f65
Oct 30 17:04:18.098486 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:18.098451    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-470be499-sw9hh" event={"ID":"aaf2e083-ca03-4482-8bc1-97d50fc52dec","Type":"ContainerStarted","Data":"e7f4710eee7e15d1eca4fe29f6421607f052c630ba370d5f4e6948b24cf60f65"}
Oct 30 17:04:19.162485 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:19.162361    4433 generic.go:356] "Generic (PLEG): container finished" podID="aaf2e083-ca03-4482-8bc1-97d50fc52dec" containerID="c2055ddc37bfaa557ac3bc0f9fc4268c5ebb44fea0b569780a161696ac9c8102" exitCode=0
Oct 30 17:04:19.169826 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:19.163014    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-470be499-sw9hh" event={"ID":"aaf2e083-ca03-4482-8bc1-97d50fc52dec","Type":"ContainerDied","Data":"c2055ddc37bfaa557ac3bc0f9fc4268c5ebb44fea0b569780a161696ac9c8102"}
Oct 30 17:04:20.377635 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:20.377571    4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:04:20.500715 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:20.490310    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" podUID="458eba2e-eb03-452a-bcf4-a5e91da83acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 17:04:20.508834 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:20.508485    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:04:20.683482 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:20.672503    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:04:20.767438 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:20.764524    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:04:20.955625 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:20.955514    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:04:20.983442 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:20.957990    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:04:21.478459 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:21.379703    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm" podUID="bf43bf3d-f126-40a1-a24f-10214a85ceb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 30 17:04:21.543429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:21.518585    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-470be499-sw9hh" event={"ID":"aaf2e083-ca03-4482-8bc1-97d50fc52dec","Type":"ContainerDied","Data":"e7f4710eee7e15d1eca4fe29f6421607f052c630ba370d5f4e6948b24cf60f65"}
Oct 30 17:04:21.543429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:21.518657    4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7f4710eee7e15d1eca4fe29f6421607f052c630ba370d5f4e6948b24cf60f65"
Oct 30 17:04:21.543429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:21.524849    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-470be499-sw9hh"
Oct 30 17:04:21.543429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:21.525250    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:04:21.543429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:21.533742    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 17:04:21.587393 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:21.549147    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" podUID="458eba2e-eb03-452a-bcf4-a5e91da83acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:04:21.592976 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:21.592929    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:04:21.594210 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:21.593497    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:04:21.594210 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:21.593574    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:04:21.657613 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:21.657526    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:04:21.674659 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:21.674615    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:04:21.684564 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:21.677529    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:04:21.963545 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:21.963492    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:04:21.965052 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:21.965016    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:04:22.001496 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:22.001457    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaf2e083-ca03-4482-8bc1-97d50fc52dec-kube-api-access-5hsc5\" (UniqueName: \"kubernetes.io/projected/aaf2e083-ca03-4482-8bc1-97d50fc52dec-kube-api-access-5hsc5\") pod \"aaf2e083-ca03-4482-8bc1-97d50fc52dec\" (UID: \"aaf2e083-ca03-4482-8bc1-97d50fc52dec\") "
Oct 30 17:04:22.011533 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:22.001996    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aaf2e083-ca03-4482-8bc1-97d50fc52dec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaf2e083-ca03-4482-8bc1-97d50fc52dec-maas-api-xks-ca\") pod \"aaf2e083-ca03-4482-8bc1-97d50fc52dec\" (UID: \"aaf2e083-ca03-4482-8bc1-97d50fc52dec\") "
Oct 30 17:04:22.044971 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:22.044937    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaf2e083-ca03-4482-8bc1-97d50fc52dec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaf2e083-ca03-4482-8bc1-97d50fc52dec-maas-api-openshift-ca\") pod \"aaf2e083-ca03-4482-8bc1-97d50fc52dec\" (UID: \"aaf2e083-ca03-4482-8bc1-97d50fc52dec\") "
Oct 30 17:04:22.049109 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:22.049042    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaf2e083-ca03-4482-8bc1-97d50fc52dec-maas-api-openshift-ca" pod "aaf2e083-ca03-4482-8bc1-97d50fc52dec" (UID: "aaf2e083-ca03-4482-8bc1-97d50fc52dec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:04:22.061128 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:22.061097    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaf2e083-ca03-4482-8bc1-97d50fc52dec-maas-api-openshift-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:04:22.090944 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:22.090917    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 17:04:22.091177 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:22.091161    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 17:04:22.117434 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:22.092516    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaf2e083-ca03-4482-8bc1-97d50fc52dec-maas-api-xks-ca" pod "aaf2e083-ca03-4482-8bc1-97d50fc52dec" (UID: "aaf2e083-ca03-4482-8bc1-97d50fc52dec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:04:22.150668 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:22.150627    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaf2e083-ca03-4482-8bc1-97d50fc52dec-kube-api-access-5hsc5" pod "aaf2e083-ca03-4482-8bc1-97d50fc52dec" (UID: "aaf2e083-ca03-4482-8bc1-97d50fc52dec"). InnerVolumeSpecName "kube-api-access-5hsc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:04:22.319848 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:22.319747    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaf2e083-ca03-4482-8bc1-97d50fc52dec-maas-api-xks-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:04:22.320080 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:22.320056    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hsc5\" (UniqueName: \"kubernetes.io/projected/aaf2e083-ca03-4482-8bc1-97d50fc52dec-kube-api-access-5hsc5\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:04:22.393108 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:22.393066    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm" podUID="bf43bf3d-f126-40a1-a24f-10214a85ceb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:04:22.412794 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:22.412762    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 17:04:22.413095 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:22.413077    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 17:04:22.572338 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:22.572233    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" podUID="458eba2e-eb03-452a-bcf4-a5e91da83acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:04:22.663430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:22.615036    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-470be499-sw9hh"
Oct 30 17:04:22.663430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:22.640111    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 17:04:22.663430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:22.640771    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 17:04:22.849822 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:22.836789    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:04:22.892630 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:22.892585    4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:04:22.903263 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:22.903229    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 17:04:22.986564 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:22.986513    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:04:23.257207 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:23.257172    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 17:04:23.281348 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:23.281316    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:04:23.321883 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:23.321835    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 17:04:23.334061 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:23.334033    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:04:26.983949 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:04:26.983909    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:04:27.055895 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:27.055857    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-295e55e4-8vsvr"]
Oct 30 17:04:27.056762 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:27.056740    4433 cpu_manager.go:414] "containerMap: removing container" podUID="aaf2e083-ca03-4482-8bc1-97d50fc52dec" containerName="revoke-keys"
Oct 30 17:04:27.056835 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:27.056766    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaf2e083-ca03-4482-8bc1-97d50fc52dec" containerName="revoke-keys"
Oct 30 17:04:27.056935 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:27.056924    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaf2e083-ca03-4482-8bc1-97d50fc52dec" containerName="revoke-keys"
Oct 30 17:04:27.061283 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:27.061241    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-295e55e4-8vsvr"
Oct 30 17:04:27.083459 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:27.083294    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-295e55e4-8vsvr"]
Oct 30 17:04:27.089323 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:27.087436    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:04:27.089323 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:27.087461    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\""
Oct 30 17:04:27.165961 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:27.165924    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f27m\" (UniqueName: \"kubernetes.io/projected/ea2c222c-a285-4a20-b69a-8c5afd89edfc-kube-api-access-9f27m\") pod \"maas-api-revoke-sub-295e55e4-8vsvr\" (UID: \"ea2c222c-a285-4a20-b69a-8c5afd89edfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-295e55e4-8vsvr"
Oct 30 17:04:27.166140 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:27.165987    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea2c222c-a285-4a20-b69a-8c5afd89edfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-295e55e4-8vsvr\" (UID: \"ea2c222c-a285-4a20-b69a-8c5afd89edfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-295e55e4-8vsvr"
Oct 30 17:04:27.166199 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:27.166177    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea2c222c-a285-4a20-b69a-8c5afd89edfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-295e55e4-8vsvr\" (UID: \"ea2c222c-a285-4a20-b69a-8c5afd89edfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-295e55e4-8vsvr"
Oct 30 17:04:27.268628 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:27.268542    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea2c222c-a285-4a20-b69a-8c5afd89edfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-295e55e4-8vsvr\" (UID: \"ea2c222c-a285-4a20-b69a-8c5afd89edfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-295e55e4-8vsvr"
Oct 30 17:04:27.268808 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:27.268661    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f27m\" (UniqueName: \"kubernetes.io/projected/ea2c222c-a285-4a20-b69a-8c5afd89edfc-kube-api-access-9f27m\") pod \"maas-api-revoke-sub-295e55e4-8vsvr\" (UID: \"ea2c222c-a285-4a20-b69a-8c5afd89edfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-295e55e4-8vsvr"
Oct 30 17:04:27.268808 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:27.268715    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea2c222c-a285-4a20-b69a-8c5afd89edfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-295e55e4-8vsvr\" (UID: \"ea2c222c-a285-4a20-b69a-8c5afd89edfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-295e55e4-8vsvr"
Oct 30 17:04:27.269328 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:27.269304    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea2c222c-a285-4a20-b69a-8c5afd89edfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-295e55e4-8vsvr\" (UID: \"ea2c222c-a285-4a20-b69a-8c5afd89edfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-295e55e4-8vsvr"
Oct 30 17:04:27.270951 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:27.270931    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea2c222c-a285-4a20-b69a-8c5afd89edfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-295e55e4-8vsvr\" (UID: \"ea2c222c-a285-4a20-b69a-8c5afd89edfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-295e55e4-8vsvr"
Oct 30 17:04:27.293901 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:27.293867    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f27m\" (UniqueName: \"kubernetes.io/projected/ea2c222c-a285-4a20-b69a-8c5afd89edfc-kube-api-access-9f27m\") pod \"maas-api-revoke-sub-295e55e4-8vsvr\" (UID: \"ea2c222c-a285-4a20-b69a-8c5afd89edfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-295e55e4-8vsvr"
Oct 30 17:04:27.387232 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:27.387195    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-295e55e4-8vsvr"
Oct 30 17:04:27.695467 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:04:27.695432    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea2c222c_a285_4a20_b69a_8c5afd89edfc.slice/crio-9cee5330a588a0c316be7b62d68caa69baadd44db15d06318d063a45768c462d WatchSource:0}: Error finding container 9cee5330a588a0c316be7b62d68caa69baadd44db15d06318d063a45768c462d: Status 404 returned error can't find the container with id 9cee5330a588a0c316be7b62d68caa69baadd44db15d06318d063a45768c462d
Oct 30 17:04:27.698455 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:27.698424    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-295e55e4-8vsvr"]
Oct 30 17:04:28.661040 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:28.660996    4433 generic.go:356] "Generic (PLEG): container finished" podID="ea2c222c-a285-4a20-b69a-8c5afd89edfc" containerID="f445d532af529a451a83433d68ed1bb8e7a1e37e7bc8c6b36977795c14bede07" exitCode=0
Oct 30 17:04:28.661540 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:28.661239    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-295e55e4-8vsvr" event={"ID":"ea2c222c-a285-4a20-b69a-8c5afd89edfc","Type":"ContainerDied","Data":"f445d532af529a451a83433d68ed1bb8e7a1e37e7bc8c6b36977795c14bede07"}
Oct 30 17:04:28.661540 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:28.661336    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-295e55e4-8vsvr" event={"ID":"ea2c222c-a285-4a20-b69a-8c5afd89edfc","Type":"ContainerStarted","Data":"9cee5330a588a0c316be7b62d68caa69baadd44db15d06318d063a45768c462d"}
Oct 30 17:04:29.836628 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:29.836565    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-295e55e4-8vsvr"
Oct 30 17:04:30.005184 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:30.005153    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea2c222c-a285-4a20-b69a-8c5afd89edfc-kube-api-access-9f27m\" (UniqueName: \"kubernetes.io/projected/ea2c222c-a285-4a20-b69a-8c5afd89edfc-kube-api-access-9f27m\") pod \"ea2c222c-a285-4a20-b69a-8c5afd89edfc\" (UID: \"ea2c222c-a285-4a20-b69a-8c5afd89edfc\") "
Oct 30 17:04:30.005378 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:30.005195    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea2c222c-a285-4a20-b69a-8c5afd89edfc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea2c222c-a285-4a20-b69a-8c5afd89edfc-maas-api-openshift-ca\") pod \"ea2c222c-a285-4a20-b69a-8c5afd89edfc\" (UID: \"ea2c222c-a285-4a20-b69a-8c5afd89edfc\") "
Oct 30 17:04:30.005378 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:30.005280    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea2c222c-a285-4a20-b69a-8c5afd89edfc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea2c222c-a285-4a20-b69a-8c5afd89edfc-maas-api-xks-ca\") pod \"ea2c222c-a285-4a20-b69a-8c5afd89edfc\" (UID: \"ea2c222c-a285-4a20-b69a-8c5afd89edfc\") "
Oct 30 17:04:30.005632 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:30.005605    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea2c222c-a285-4a20-b69a-8c5afd89edfc-maas-api-openshift-ca" pod "ea2c222c-a285-4a20-b69a-8c5afd89edfc" (UID: "ea2c222c-a285-4a20-b69a-8c5afd89edfc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:04:30.005869 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:30.005851    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea2c222c-a285-4a20-b69a-8c5afd89edfc-maas-api-openshift-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:04:30.007417 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:30.007375    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea2c222c-a285-4a20-b69a-8c5afd89edfc-maas-api-xks-ca" pod "ea2c222c-a285-4a20-b69a-8c5afd89edfc" (UID: "ea2c222c-a285-4a20-b69a-8c5afd89edfc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:04:30.007417 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:30.007386    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea2c222c-a285-4a20-b69a-8c5afd89edfc-kube-api-access-9f27m" pod "ea2c222c-a285-4a20-b69a-8c5afd89edfc" (UID: "ea2c222c-a285-4a20-b69a-8c5afd89edfc"). InnerVolumeSpecName "kube-api-access-9f27m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:04:30.106937 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:30.106852    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f27m\" (UniqueName: \"kubernetes.io/projected/ea2c222c-a285-4a20-b69a-8c5afd89edfc-kube-api-access-9f27m\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:04:30.106937 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:30.106880    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea2c222c-a285-4a20-b69a-8c5afd89edfc-maas-api-xks-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:04:30.675104 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:30.675078    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-295e55e4-8vsvr"
Oct 30 17:04:30.675278 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:30.675111    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-295e55e4-8vsvr" event={"ID":"ea2c222c-a285-4a20-b69a-8c5afd89edfc","Type":"ContainerDied","Data":"9cee5330a588a0c316be7b62d68caa69baadd44db15d06318d063a45768c462d"}
Oct 30 17:04:30.675278 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:30.675169    4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cee5330a588a0c316be7b62d68caa69baadd44db15d06318d063a45768c462d"
Oct 30 17:04:38.437247 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:38.437210    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f284e553-qvdvd"]
Oct 30 17:04:38.438053 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:38.438026    4433 cpu_manager.go:414] "containerMap: removing container" podUID="ea2c222c-a285-4a20-b69a-8c5afd89edfc" containerName="revoke-keys"
Oct 30 17:04:38.438053 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:38.438052    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea2c222c-a285-4a20-b69a-8c5afd89edfc" containerName="revoke-keys"
Oct 30 17:04:38.438248 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:38.438200    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea2c222c-a285-4a20-b69a-8c5afd89edfc" containerName="revoke-keys"
Oct 30 17:04:38.442261 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:38.442241    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f284e553-qvdvd"
Oct 30 17:04:38.456696 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:38.456674    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:04:38.474445 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:38.467997    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\""
Oct 30 17:04:38.474445 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:38.474309    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f284e553-qvdvd"]
Oct 30 17:04:38.530859 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:38.530821    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70440f57-cc90-47c9-8845-b69d35c82cbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f284e553-qvdvd\" (UID: \"70440f57-cc90-47c9-8845-b69d35c82cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f284e553-qvdvd"
Oct 30 17:04:38.531022 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:38.530864    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70440f57-cc90-47c9-8845-b69d35c82cbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f284e553-qvdvd\" (UID: \"70440f57-cc90-47c9-8845-b69d35c82cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f284e553-qvdvd"
Oct 30 17:04:38.531127 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:38.531113    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrcwn\" (UniqueName: \"kubernetes.io/projected/70440f57-cc90-47c9-8845-b69d35c82cbc-kube-api-access-xrcwn\") pod \"maas-api-revoke-sub-f284e553-qvdvd\" (UID: \"70440f57-cc90-47c9-8845-b69d35c82cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f284e553-qvdvd"
Oct 30 17:04:38.632969 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:38.632930    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrcwn\" (UniqueName: \"kubernetes.io/projected/70440f57-cc90-47c9-8845-b69d35c82cbc-kube-api-access-xrcwn\") pod \"maas-api-revoke-sub-f284e553-qvdvd\" (UID: \"70440f57-cc90-47c9-8845-b69d35c82cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f284e553-qvdvd"
Oct 30 17:04:38.633170 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:38.633024    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70440f57-cc90-47c9-8845-b69d35c82cbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f284e553-qvdvd\" (UID: \"70440f57-cc90-47c9-8845-b69d35c82cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f284e553-qvdvd"
Oct 30 17:04:38.633170 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:38.633060    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70440f57-cc90-47c9-8845-b69d35c82cbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f284e553-qvdvd\" (UID: \"70440f57-cc90-47c9-8845-b69d35c82cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f284e553-qvdvd"
Oct 30 17:04:38.633855 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:38.633831    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70440f57-cc90-47c9-8845-b69d35c82cbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f284e553-qvdvd\" (UID: \"70440f57-cc90-47c9-8845-b69d35c82cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f284e553-qvdvd"
Oct 30 17:04:38.635570 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:38.635548    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70440f57-cc90-47c9-8845-b69d35c82cbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f284e553-qvdvd\" (UID: \"70440f57-cc90-47c9-8845-b69d35c82cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f284e553-qvdvd"
Oct 30 17:04:38.655885 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:38.655847    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrcwn\" (UniqueName: \"kubernetes.io/projected/70440f57-cc90-47c9-8845-b69d35c82cbc-kube-api-access-xrcwn\") pod \"maas-api-revoke-sub-f284e553-qvdvd\" (UID: \"70440f57-cc90-47c9-8845-b69d35c82cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f284e553-qvdvd"
Oct 30 17:04:38.756859 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:38.756776    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f284e553-qvdvd"
Oct 30 17:04:39.066821 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:39.066742    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f284e553-qvdvd"]
Oct 30 17:04:39.080747 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:04:39.080714    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70440f57_cc90_47c9_8845_b69d35c82cbc.slice/crio-57359b62d9e7182b9fb9fd49ad88b3c606ce18a470db48058c27eeb1f800653f WatchSource:0}: Error finding container 57359b62d9e7182b9fb9fd49ad88b3c606ce18a470db48058c27eeb1f800653f: Status 404 returned error can't find the container with id 57359b62d9e7182b9fb9fd49ad88b3c606ce18a470db48058c27eeb1f800653f
Oct 30 17:04:39.734124 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:39.734077    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f284e553-qvdvd" event={"ID":"70440f57-cc90-47c9-8845-b69d35c82cbc","Type":"ContainerStarted","Data":"b60edfa237cec9c672a22af128c6e559584fbe7476ed50046717990a03236737"}
Oct 30 17:04:39.734588 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:39.734146    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f284e553-qvdvd" event={"ID":"70440f57-cc90-47c9-8845-b69d35c82cbc","Type":"ContainerStarted","Data":"57359b62d9e7182b9fb9fd49ad88b3c606ce18a470db48058c27eeb1f800653f"}
Oct 30 17:04:39.775221 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:39.775167    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f284e553-qvdvd" podStartSLOduration=1.775151314 podStartE2EDuration="1.775151314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:04:39.772936381 +0000 UTC m=+1056.207838602" watchObservedRunningTime="2026-10-30 17:04:39.775151314 +0000 UTC m=+1056.210053519"
Oct 30 17:04:40.063325 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:40.063230    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:04:40.753822 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:40.753778    4433 generic.go:356] "Generic (PLEG): container finished" podID="70440f57-cc90-47c9-8845-b69d35c82cbc" containerID="b60edfa237cec9c672a22af128c6e559584fbe7476ed50046717990a03236737" exitCode=0
Oct 30 17:04:40.754279 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:40.753865    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f284e553-qvdvd" event={"ID":"70440f57-cc90-47c9-8845-b69d35c82cbc","Type":"ContainerDied","Data":"b60edfa237cec9c672a22af128c6e559584fbe7476ed50046717990a03236737"}
Oct 30 17:04:41.164096 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:41.164006    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99ada05d-hhmhw"]
Oct 30 17:04:41.168701 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:41.168671    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99ada05d-hhmhw"
Oct 30 17:04:41.201468 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:41.194979    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99ada05d-hhmhw"]
Oct 30 17:04:41.280502 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:41.280460    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7afb8ffd-67e2-40e0-89e3-d2cb40667682-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99ada05d-hhmhw\" (UID: \"7afb8ffd-67e2-40e0-89e3-d2cb40667682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99ada05d-hhmhw"
Oct 30 17:04:41.280698 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:41.280524    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmwr5\" (UniqueName: \"kubernetes.io/projected/7afb8ffd-67e2-40e0-89e3-d2cb40667682-kube-api-access-fmwr5\") pod \"maas-api-revoke-sub-99ada05d-hhmhw\" (UID: \"7afb8ffd-67e2-40e0-89e3-d2cb40667682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99ada05d-hhmhw"
Oct 30 17:04:41.280777 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:41.280693    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7afb8ffd-67e2-40e0-89e3-d2cb40667682-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99ada05d-hhmhw\" (UID: \"7afb8ffd-67e2-40e0-89e3-d2cb40667682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99ada05d-hhmhw"
Oct 30 17:04:41.382431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:41.382090    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7afb8ffd-67e2-40e0-89e3-d2cb40667682-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99ada05d-hhmhw\" (UID: \"7afb8ffd-67e2-40e0-89e3-d2cb40667682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99ada05d-hhmhw"
Oct 30 17:04:41.382431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:41.382159    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmwr5\" (UniqueName: \"kubernetes.io/projected/7afb8ffd-67e2-40e0-89e3-d2cb40667682-kube-api-access-fmwr5\") pod \"maas-api-revoke-sub-99ada05d-hhmhw\" (UID: \"7afb8ffd-67e2-40e0-89e3-d2cb40667682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99ada05d-hhmhw"
Oct 30 17:04:41.382431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:41.382256    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7afb8ffd-67e2-40e0-89e3-d2cb40667682-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99ada05d-hhmhw\" (UID: \"7afb8ffd-67e2-40e0-89e3-d2cb40667682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99ada05d-hhmhw"
Oct 30 17:04:41.386457 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:41.383257    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7afb8ffd-67e2-40e0-89e3-d2cb40667682-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99ada05d-hhmhw\" (UID: \"7afb8ffd-67e2-40e0-89e3-d2cb40667682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99ada05d-hhmhw"
Oct 30 17:04:41.388024 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:41.387966    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7afb8ffd-67e2-40e0-89e3-d2cb40667682-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99ada05d-hhmhw\" (UID: \"7afb8ffd-67e2-40e0-89e3-d2cb40667682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99ada05d-hhmhw"
Oct 30 17:04:41.409091 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:41.409058    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmwr5\" (UniqueName: \"kubernetes.io/projected/7afb8ffd-67e2-40e0-89e3-d2cb40667682-kube-api-access-fmwr5\") pod \"maas-api-revoke-sub-99ada05d-hhmhw\" (UID: \"7afb8ffd-67e2-40e0-89e3-d2cb40667682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99ada05d-hhmhw"
Oct 30 17:04:41.493003 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:41.492615    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99ada05d-hhmhw"
Oct 30 17:04:42.741496 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:42.693630    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:04:42.786476 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:42.779504    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:04:42.826146 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:42.826100    4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:04:42.857049 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:42.830921    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99ada05d-hhmhw"]
Oct 30 17:04:42.884366 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:04:42.884313    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7afb8ffd_67e2_40e0_89e3_d2cb40667682.slice/crio-9ac97c77b02c4fe59429a3d9f988f4edb7a2e581dc4f846bbe842eef320c9fd4 WatchSource:0}: Error finding container 9ac97c77b02c4fe59429a3d9f988f4edb7a2e581dc4f846bbe842eef320c9fd4: Status 404 returned error can't find the container with id 9ac97c77b02c4fe59429a3d9f988f4edb7a2e581dc4f846bbe842eef320c9fd4
Oct 30 17:04:42.910938 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:42.910902    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 17:04:43.018191 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:43.014053    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:04:43.140167 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:43.140129    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99ada05d-hhmhw" event={"ID":"7afb8ffd-67e2-40e0-89e3-d2cb40667682","Type":"ContainerStarted","Data":"9ac97c77b02c4fe59429a3d9f988f4edb7a2e581dc4f846bbe842eef320c9fd4"}
Oct 30 17:04:43.299516 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:43.299483    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f284e553-qvdvd"
Oct 30 17:04:43.322429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:43.320569    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" podUID="458eba2e-eb03-452a-bcf4-a5e91da83acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 17:04:43.382430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:43.382086    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70440f57-cc90-47c9-8845-b69d35c82cbc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70440f57-cc90-47c9-8845-b69d35c82cbc-maas-api-openshift-ca\") pod \"70440f57-cc90-47c9-8845-b69d35c82cbc\" (UID: \"70440f57-cc90-47c9-8845-b69d35c82cbc\") "
Oct 30 17:04:43.382430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:43.382387    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70440f57-cc90-47c9-8845-b69d35c82cbc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70440f57-cc90-47c9-8845-b69d35c82cbc-maas-api-xks-ca\") pod \"70440f57-cc90-47c9-8845-b69d35c82cbc\" (UID: \"70440f57-cc90-47c9-8845-b69d35c82cbc\") "
Oct 30 17:04:43.382699 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:43.382453    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70440f57-cc90-47c9-8845-b69d35c82cbc-kube-api-access-xrcwn\" (UniqueName: \"kubernetes.io/projected/70440f57-cc90-47c9-8845-b69d35c82cbc-kube-api-access-xrcwn\") pod \"70440f57-cc90-47c9-8845-b69d35c82cbc\" (UID: \"70440f57-cc90-47c9-8845-b69d35c82cbc\") "
Oct 30 17:04:43.386429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:43.384621    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70440f57-cc90-47c9-8845-b69d35c82cbc-maas-api-openshift-ca" pod "70440f57-cc90-47c9-8845-b69d35c82cbc" (UID: "70440f57-cc90-47c9-8845-b69d35c82cbc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:04:43.416639 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:43.416580    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70440f57-cc90-47c9-8845-b69d35c82cbc-maas-api-xks-ca" pod "70440f57-cc90-47c9-8845-b69d35c82cbc" (UID: "70440f57-cc90-47c9-8845-b69d35c82cbc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:04:43.421603 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:43.421568    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70440f57-cc90-47c9-8845-b69d35c82cbc-kube-api-access-xrcwn" pod "70440f57-cc90-47c9-8845-b69d35c82cbc" (UID: "70440f57-cc90-47c9-8845-b69d35c82cbc"). InnerVolumeSpecName "kube-api-access-xrcwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:04:43.433743 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:43.433620    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:04:43.490767 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:43.490668    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70440f57-cc90-47c9-8845-b69d35c82cbc-maas-api-xks-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:04:43.490767 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:43.490709    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrcwn\" (UniqueName: \"kubernetes.io/projected/70440f57-cc90-47c9-8845-b69d35c82cbc-kube-api-access-xrcwn\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:04:43.490767 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:43.490725    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70440f57-cc90-47c9-8845-b69d35c82cbc-maas-api-openshift-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:04:43.491431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:43.491310    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:04:43.495984 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:43.495950    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:04:44.033903 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:44.033813    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 17:04:44.331428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:44.320148    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm" podUID="bf43bf3d-f126-40a1-a24f-10214a85ceb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:04:44.357624 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:44.357575    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" podUID="458eba2e-eb03-452a-bcf4-a5e91da83acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 17:04:44.434672 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:44.434623    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:04:45.103079 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:04:45.103045    4433 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.415s"
Oct 30 17:04:45.217098 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:45.217056    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 17:04:45.217673 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:45.217652    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:04:45.218762 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:45.218745    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:04:45.219607 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:45.219592    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 17:04:45.220472 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:45.220393    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:04:45.220689 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:45.220673    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:04:45.235383 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:45.235343    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f284e553-qvdvd" event={"ID":"70440f57-cc90-47c9-8845-b69d35c82cbc","Type":"ContainerDied","Data":"57359b62d9e7182b9fb9fd49ad88b3c606ce18a470db48058c27eeb1f800653f"}
Oct 30 17:04:45.236025 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:45.236008    4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57359b62d9e7182b9fb9fd49ad88b3c606ce18a470db48058c27eeb1f800653f"
Oct 30 17:04:45.236162 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:45.235931    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f284e553-qvdvd"
Oct 30 17:04:45.809432 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:45.793572    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 17:04:45.948572 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:45.948524    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:04:46.243896 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:46.243857    4433 generic.go:356] "Generic (PLEG): container finished" podID="7afb8ffd-67e2-40e0-89e3-d2cb40667682" containerID="138fe3a84b4e56e7c224dec77329a6e7788027414a2ed606fb1b596df70c1039" exitCode=0
Oct 30 17:04:46.244386 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:46.244008    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99ada05d-hhmhw" event={"ID":"7afb8ffd-67e2-40e0-89e3-d2cb40667682","Type":"ContainerDied","Data":"138fe3a84b4e56e7c224dec77329a6e7788027414a2ed606fb1b596df70c1039"}
Oct 30 17:04:47.417901 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:47.417873    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99ada05d-hhmhw"
Oct 30 17:04:47.539711 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:47.539623    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7afb8ffd-67e2-40e0-89e3-d2cb40667682-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7afb8ffd-67e2-40e0-89e3-d2cb40667682-maas-api-openshift-ca\") pod \"7afb8ffd-67e2-40e0-89e3-d2cb40667682\" (UID: \"7afb8ffd-67e2-40e0-89e3-d2cb40667682\") "
Oct 30 17:04:47.539711 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:47.539696    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7afb8ffd-67e2-40e0-89e3-d2cb40667682-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7afb8ffd-67e2-40e0-89e3-d2cb40667682-maas-api-xks-ca\") pod \"7afb8ffd-67e2-40e0-89e3-d2cb40667682\" (UID: \"7afb8ffd-67e2-40e0-89e3-d2cb40667682\") "
Oct 30 17:04:47.539944 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:47.539830    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7afb8ffd-67e2-40e0-89e3-d2cb40667682-kube-api-access-fmwr5\" (UniqueName: \"kubernetes.io/projected/7afb8ffd-67e2-40e0-89e3-d2cb40667682-kube-api-access-fmwr5\") pod \"7afb8ffd-67e2-40e0-89e3-d2cb40667682\" (UID: \"7afb8ffd-67e2-40e0-89e3-d2cb40667682\") "
Oct 30 17:04:47.540091 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:47.540050    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7afb8ffd-67e2-40e0-89e3-d2cb40667682-maas-api-openshift-ca" pod "7afb8ffd-67e2-40e0-89e3-d2cb40667682" (UID: "7afb8ffd-67e2-40e0-89e3-d2cb40667682"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:04:47.540554 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:47.540528    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7afb8ffd-67e2-40e0-89e3-d2cb40667682-maas-api-openshift-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:04:47.541926 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:47.541897    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7afb8ffd-67e2-40e0-89e3-d2cb40667682-maas-api-xks-ca" pod "7afb8ffd-67e2-40e0-89e3-d2cb40667682" (UID: "7afb8ffd-67e2-40e0-89e3-d2cb40667682"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:04:47.542049 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:47.542029    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7afb8ffd-67e2-40e0-89e3-d2cb40667682-kube-api-access-fmwr5" pod "7afb8ffd-67e2-40e0-89e3-d2cb40667682" (UID: "7afb8ffd-67e2-40e0-89e3-d2cb40667682"). InnerVolumeSpecName "kube-api-access-fmwr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:04:47.642343 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:47.642308    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7afb8ffd-67e2-40e0-89e3-d2cb40667682-maas-api-xks-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:04:47.642343 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:47.642337    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmwr5\" (UniqueName: \"kubernetes.io/projected/7afb8ffd-67e2-40e0-89e3-d2cb40667682-kube-api-access-fmwr5\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:04:48.258017 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:48.257983    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99ada05d-hhmhw"
Oct 30 17:04:48.258273 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:48.258017    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99ada05d-hhmhw" event={"ID":"7afb8ffd-67e2-40e0-89e3-d2cb40667682","Type":"ContainerDied","Data":"9ac97c77b02c4fe59429a3d9f988f4edb7a2e581dc4f846bbe842eef320c9fd4"}
Oct 30 17:04:48.258273 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:48.258094    4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ac97c77b02c4fe59429a3d9f988f4edb7a2e581dc4f846bbe842eef320c9fd4"
Oct 30 17:04:48.798999 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:48.798965    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2e94962-v57p5"]
Oct 30 17:04:48.799559 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:48.799541    4433 cpu_manager.go:414] "containerMap: removing container" podUID="70440f57-cc90-47c9-8845-b69d35c82cbc" containerName="revoke-keys"
Oct 30 17:04:48.799611 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:48.799561    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70440f57-cc90-47c9-8845-b69d35c82cbc" containerName="revoke-keys"
Oct 30 17:04:48.799611 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:48.799574    4433 cpu_manager.go:414] "containerMap: removing container" podUID="7afb8ffd-67e2-40e0-89e3-d2cb40667682" containerName="revoke-keys"
Oct 30 17:04:48.799611 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:48.799579    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7afb8ffd-67e2-40e0-89e3-d2cb40667682" containerName="revoke-keys"
Oct 30 17:04:48.799707 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:48.799687    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="70440f57-cc90-47c9-8845-b69d35c82cbc" containerName="revoke-keys"
Oct 30 17:04:48.799707 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:48.799696    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="7afb8ffd-67e2-40e0-89e3-d2cb40667682" containerName="revoke-keys"
Oct 30 17:04:48.803267 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:48.803248    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e94962-v57p5"
Oct 30 17:04:48.820453 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:48.819561    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:04:48.820453 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:48.820227    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\""
Oct 30 17:04:48.823361 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:48.823331    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2e94962-v57p5"]
Oct 30 17:04:48.860983 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:48.860953    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mflsz\" (UniqueName: \"kubernetes.io/projected/80030931-275e-44e9-8f11-702f7c863213-kube-api-access-mflsz\") pod \"maas-api-revoke-sub-c2e94962-v57p5\" (UID: \"80030931-275e-44e9-8f11-702f7c863213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e94962-v57p5"
Oct 30 17:04:48.861237 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:48.861213    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80030931-275e-44e9-8f11-702f7c863213-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2e94962-v57p5\" (UID: \"80030931-275e-44e9-8f11-702f7c863213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e94962-v57p5"
Oct 30 17:04:48.861357 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:48.861336    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80030931-275e-44e9-8f11-702f7c863213-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2e94962-v57p5\" (UID: \"80030931-275e-44e9-8f11-702f7c863213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e94962-v57p5"
Oct 30 17:04:48.962233 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:48.962182    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80030931-275e-44e9-8f11-702f7c863213-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2e94962-v57p5\" (UID: \"80030931-275e-44e9-8f11-702f7c863213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e94962-v57p5"
Oct 30 17:04:48.962486 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:48.962272    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80030931-275e-44e9-8f11-702f7c863213-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2e94962-v57p5\" (UID: \"80030931-275e-44e9-8f11-702f7c863213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e94962-v57p5"
Oct 30 17:04:48.962486 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:48.962335    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mflsz\" (UniqueName: \"kubernetes.io/projected/80030931-275e-44e9-8f11-702f7c863213-kube-api-access-mflsz\") pod \"maas-api-revoke-sub-c2e94962-v57p5\" (UID: \"80030931-275e-44e9-8f11-702f7c863213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e94962-v57p5"
Oct 30 17:04:48.962911 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:48.962886    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80030931-275e-44e9-8f11-702f7c863213-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2e94962-v57p5\" (UID: \"80030931-275e-44e9-8f11-702f7c863213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e94962-v57p5"
Oct 30 17:04:48.964717 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:48.964698    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80030931-275e-44e9-8f11-702f7c863213-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2e94962-v57p5\" (UID: \"80030931-275e-44e9-8f11-702f7c863213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e94962-v57p5"
Oct 30 17:04:48.985377 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:48.985346    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mflsz\" (UniqueName: \"kubernetes.io/projected/80030931-275e-44e9-8f11-702f7c863213-kube-api-access-mflsz\") pod \"maas-api-revoke-sub-c2e94962-v57p5\" (UID: \"80030931-275e-44e9-8f11-702f7c863213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e94962-v57p5"
Oct 30 17:04:49.119642 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:49.119566    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e94962-v57p5"
Oct 30 17:04:49.424636 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:49.424605    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2e94962-v57p5"]
Oct 30 17:04:49.425739 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:04:49.425712    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80030931_275e_44e9_8f11_702f7c863213.slice/crio-cf612aab2e547fe76f456472c0ce409b883b49d0f29a110e05ffddaf27be38f4 WatchSource:0}: Error finding container cf612aab2e547fe76f456472c0ce409b883b49d0f29a110e05ffddaf27be38f4: Status 404 returned error can't find the container with id cf612aab2e547fe76f456472c0ce409b883b49d0f29a110e05ffddaf27be38f4
Oct 30 17:04:50.273749 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:50.273716    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e94962-v57p5" event={"ID":"80030931-275e-44e9-8f11-702f7c863213","Type":"ContainerStarted","Data":"009c76a50730ac358707b28576f6ed70f5dba7ab2cf007df3151e69b1e52c1ef"}
Oct 30 17:04:50.274144 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:50.273795    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e94962-v57p5" event={"ID":"80030931-275e-44e9-8f11-702f7c863213","Type":"ContainerStarted","Data":"cf612aab2e547fe76f456472c0ce409b883b49d0f29a110e05ffddaf27be38f4"}
Oct 30 17:04:50.319361 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:50.319280    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e94962-v57p5" podStartSLOduration=2.319261458 podStartE2EDuration="2.319261458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:04:50.312536283 +0000 UTC m=+1066.747438487" watchObservedRunningTime="2026-10-30 17:04:50.319261458 +0000 UTC m=+1066.754163665"
Oct 30 17:04:51.281984 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:51.281944    4433 generic.go:356] "Generic (PLEG): container finished" podID="80030931-275e-44e9-8f11-702f7c863213" containerID="009c76a50730ac358707b28576f6ed70f5dba7ab2cf007df3151e69b1e52c1ef" exitCode=0
Oct 30 17:04:51.282376 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:51.282040    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e94962-v57p5" event={"ID":"80030931-275e-44e9-8f11-702f7c863213","Type":"ContainerDied","Data":"009c76a50730ac358707b28576f6ed70f5dba7ab2cf007df3151e69b1e52c1ef"}
Oct 30 17:04:52.451688 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:52.451663    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e94962-v57p5"
Oct 30 17:04:52.509370 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:52.509332    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80030931-275e-44e9-8f11-702f7c863213-kube-api-access-mflsz\" (UniqueName: \"kubernetes.io/projected/80030931-275e-44e9-8f11-702f7c863213-kube-api-access-mflsz\") pod \"80030931-275e-44e9-8f11-702f7c863213\" (UID: \"80030931-275e-44e9-8f11-702f7c863213\") "
Oct 30 17:04:52.509634 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:52.509396    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80030931-275e-44e9-8f11-702f7c863213-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80030931-275e-44e9-8f11-702f7c863213-maas-api-xks-ca\") pod \"80030931-275e-44e9-8f11-702f7c863213\" (UID: \"80030931-275e-44e9-8f11-702f7c863213\") "
Oct 30 17:04:52.509705 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:52.509665    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80030931-275e-44e9-8f11-702f7c863213-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80030931-275e-44e9-8f11-702f7c863213-maas-api-openshift-ca\") pod \"80030931-275e-44e9-8f11-702f7c863213\" (UID: \"80030931-275e-44e9-8f11-702f7c863213\") "
Oct 30 17:04:52.510920 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:52.510891    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80030931-275e-44e9-8f11-702f7c863213-maas-api-openshift-ca" pod "80030931-275e-44e9-8f11-702f7c863213" (UID: "80030931-275e-44e9-8f11-702f7c863213"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:04:52.513296 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:52.513254    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80030931-275e-44e9-8f11-702f7c863213-kube-api-access-mflsz" pod "80030931-275e-44e9-8f11-702f7c863213" (UID: "80030931-275e-44e9-8f11-702f7c863213"). InnerVolumeSpecName "kube-api-access-mflsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:04:52.513649 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:52.513623    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80030931-275e-44e9-8f11-702f7c863213-maas-api-xks-ca" pod "80030931-275e-44e9-8f11-702f7c863213" (UID: "80030931-275e-44e9-8f11-702f7c863213"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:04:52.611866 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:52.611782    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mflsz\" (UniqueName: \"kubernetes.io/projected/80030931-275e-44e9-8f11-702f7c863213-kube-api-access-mflsz\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:04:52.611866 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:52.611811    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80030931-275e-44e9-8f11-702f7c863213-maas-api-xks-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:04:52.611866 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:52.611830    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80030931-275e-44e9-8f11-702f7c863213-maas-api-openshift-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:04:53.298498 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:53.298464    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e94962-v57p5"
Oct 30 17:04:53.298679 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:53.298473    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e94962-v57p5" event={"ID":"80030931-275e-44e9-8f11-702f7c863213","Type":"ContainerDied","Data":"cf612aab2e547fe76f456472c0ce409b883b49d0f29a110e05ffddaf27be38f4"}
Oct 30 17:04:53.298679 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:04:53.298599    4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf612aab2e547fe76f456472c0ce409b883b49d0f29a110e05ffddaf27be38f4"
Oct 30 17:04:56.978475 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:04:56.978435    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:05:00.283435 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:00.281530    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" podUID="458eba2e-eb03-452a-bcf4-a5e91da83acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:00.312554 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:00.312512    4433 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:05:00.312835 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:00.312803    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6" podUID="c690570f-1cba-450b-bcf6-0549e1ded4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:00.356154 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:00.356110    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:00.383890 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:00.383477    4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:00.404745 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:00.404698    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:00.491561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:00.491508    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:00.679490 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:00.675510    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 17:05:00.785427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:00.784543    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:00.913424 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:00.909485    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:00.993936 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:00.993809    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:01.283608 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:01.283505    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" podUID="458eba2e-eb03-452a-bcf4-a5e91da83acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:01.284892 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:01.284865    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:05:01.486431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:01.474496    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 30 17:05:01.486431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:01.475323    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 17:05:01.486993 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:01.486961    4433 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:05:01.487259 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:01.487217    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz" podUID="7dded80b-9e71-4566-944c-4d548c59a684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:01.518265 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:01.492523    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:01.518265 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:01.492838    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 17:05:01.668520 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:01.667677    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 17:05:01.718990 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:01.717864    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:01.925616 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:01.925147    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 17:05:01.948376 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:01.948341    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:05:01.973571 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:01.973538    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 17:05:01.973951 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:01.973928    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:05:02.009124 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:02.007177    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:05:02.047421 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:02.037850    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 17:05:02.071218 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:02.071177    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 17:05:02.326287 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:02.325844    4433 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:05:02.326287 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:02.325912    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9" podUID="41ff63dc-b547-4143-adb5-d904bc6e91d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:17.115304 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:17.115245    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-873a1264-xlgxz"]
Oct 30 17:05:17.116359 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:17.116335    4433 cpu_manager.go:414] "containerMap: removing container" podUID="80030931-275e-44e9-8f11-702f7c863213" containerName="revoke-keys"
Oct 30 17:05:17.116359 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:17.116359    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80030931-275e-44e9-8f11-702f7c863213" containerName="revoke-keys"
Oct 30 17:05:17.117352 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:17.117318    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="80030931-275e-44e9-8f11-702f7c863213" containerName="revoke-keys"
Oct 30 17:05:17.127072 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:17.127046    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a1264-xlgxz"
Oct 30 17:05:17.142025 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:17.141991    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-873a1264-xlgxz"]
Oct 30 17:05:17.192516 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:17.192202    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:05:17.192516 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:17.192387    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\""
Oct 30 17:05:17.268826 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:17.268247    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt2wf\" (UniqueName: \"kubernetes.io/projected/f15feeeb-5e97-4e0e-935a-553affd91bb1-kube-api-access-vt2wf\") pod \"maas-api-revoke-sub-873a1264-xlgxz\" (UID: \"f15feeeb-5e97-4e0e-935a-553affd91bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a1264-xlgxz"
Oct 30 17:05:17.268826 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:17.268433    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f15feeeb-5e97-4e0e-935a-553affd91bb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-873a1264-xlgxz\" (UID: \"f15feeeb-5e97-4e0e-935a-553affd91bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a1264-xlgxz"
Oct 30 17:05:17.268826 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:17.268485    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f15feeeb-5e97-4e0e-935a-553affd91bb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-873a1264-xlgxz\" (UID: \"f15feeeb-5e97-4e0e-935a-553affd91bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a1264-xlgxz"
Oct 30 17:05:17.371298 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:17.371210    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f15feeeb-5e97-4e0e-935a-553affd91bb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-873a1264-xlgxz\" (UID: \"f15feeeb-5e97-4e0e-935a-553affd91bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a1264-xlgxz"
Oct 30 17:05:17.371298 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:17.371262    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f15feeeb-5e97-4e0e-935a-553affd91bb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-873a1264-xlgxz\" (UID: \"f15feeeb-5e97-4e0e-935a-553affd91bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a1264-xlgxz"
Oct 30 17:05:17.372756 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:17.372425    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f15feeeb-5e97-4e0e-935a-553affd91bb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-873a1264-xlgxz\" (UID: \"f15feeeb-5e97-4e0e-935a-553affd91bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a1264-xlgxz"
Oct 30 17:05:17.372756 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:17.372568    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt2wf\" (UniqueName: \"kubernetes.io/projected/f15feeeb-5e97-4e0e-935a-553affd91bb1-kube-api-access-vt2wf\") pod \"maas-api-revoke-sub-873a1264-xlgxz\" (UID: \"f15feeeb-5e97-4e0e-935a-553affd91bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a1264-xlgxz"
Oct 30 17:05:17.375180 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:17.375099    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f15feeeb-5e97-4e0e-935a-553affd91bb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-873a1264-xlgxz\" (UID: \"f15feeeb-5e97-4e0e-935a-553affd91bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a1264-xlgxz"
Oct 30 17:05:17.396461 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:17.396419    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt2wf\" (UniqueName: \"kubernetes.io/projected/f15feeeb-5e97-4e0e-935a-553affd91bb1-kube-api-access-vt2wf\") pod \"maas-api-revoke-sub-873a1264-xlgxz\" (UID: \"f15feeeb-5e97-4e0e-935a-553affd91bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a1264-xlgxz"
Oct 30 17:05:17.461431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:17.456458    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a1264-xlgxz"
Oct 30 17:05:17.920830 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:17.919051    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-873a1264-xlgxz"]
Oct 30 17:05:18.676037 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:18.675999    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a1264-xlgxz" event={"ID":"f15feeeb-5e97-4e0e-935a-553affd91bb1","Type":"ContainerStarted","Data":"b0cfddaf6430b3491e3671650af52dcb32bc7809de1f93fb1e46b7a7392e786f"}
Oct 30 17:05:18.676555 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:18.676057    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a1264-xlgxz" event={"ID":"f15feeeb-5e97-4e0e-935a-553affd91bb1","Type":"ContainerStarted","Data":"912ac33966482884fd1fb11fb2730c219d6341879436f5348420c570430c0319"}
Oct 30 17:05:18.719752 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:18.719686    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a1264-xlgxz" podStartSLOduration=1.719666911 podStartE2EDuration="1.719666911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:18.715712401 +0000 UTC m=+1095.150614606" watchObservedRunningTime="2026-10-30 17:05:18.719666911 +0000 UTC m=+1095.154569122"
Oct 30 17:05:19.685421 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:19.685366    4433 generic.go:356] "Generic (PLEG): container finished" podID="f15feeeb-5e97-4e0e-935a-553affd91bb1" containerID="b0cfddaf6430b3491e3671650af52dcb32bc7809de1f93fb1e46b7a7392e786f" exitCode=0
Oct 30 17:05:19.685868 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:19.685472    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a1264-xlgxz" event={"ID":"f15feeeb-5e97-4e0e-935a-553affd91bb1","Type":"ContainerDied","Data":"b0cfddaf6430b3491e3671650af52dcb32bc7809de1f93fb1e46b7a7392e786f"}
Oct 30 17:05:20.856905 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:20.856878    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a1264-xlgxz"
Oct 30 17:05:21.020093 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:21.019997    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f15feeeb-5e97-4e0e-935a-553affd91bb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f15feeeb-5e97-4e0e-935a-553affd91bb1-maas-api-xks-ca\") pod \"f15feeeb-5e97-4e0e-935a-553affd91bb1\" (UID: \"f15feeeb-5e97-4e0e-935a-553affd91bb1\") "
Oct 30 17:05:21.020093 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:21.020077    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f15feeeb-5e97-4e0e-935a-553affd91bb1-kube-api-access-vt2wf\" (UniqueName: \"kubernetes.io/projected/f15feeeb-5e97-4e0e-935a-553affd91bb1-kube-api-access-vt2wf\") pod \"f15feeeb-5e97-4e0e-935a-553affd91bb1\" (UID: \"f15feeeb-5e97-4e0e-935a-553affd91bb1\") "
Oct 30 17:05:21.020323 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:21.020130    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f15feeeb-5e97-4e0e-935a-553affd91bb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f15feeeb-5e97-4e0e-935a-553affd91bb1-maas-api-openshift-ca\") pod \"f15feeeb-5e97-4e0e-935a-553affd91bb1\" (UID: \"f15feeeb-5e97-4e0e-935a-553affd91bb1\") "
Oct 30 17:05:21.020679 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:21.020655    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f15feeeb-5e97-4e0e-935a-553affd91bb1-maas-api-openshift-ca" pod "f15feeeb-5e97-4e0e-935a-553affd91bb1" (UID: "f15feeeb-5e97-4e0e-935a-553affd91bb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:05:21.022572 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:21.022542    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f15feeeb-5e97-4e0e-935a-553affd91bb1-kube-api-access-vt2wf" pod "f15feeeb-5e97-4e0e-935a-553affd91bb1" (UID: "f15feeeb-5e97-4e0e-935a-553affd91bb1"). InnerVolumeSpecName "kube-api-access-vt2wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:05:21.023660 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:21.023637    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f15feeeb-5e97-4e0e-935a-553affd91bb1-maas-api-xks-ca" pod "f15feeeb-5e97-4e0e-935a-553affd91bb1" (UID: "f15feeeb-5e97-4e0e-935a-553affd91bb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:05:21.122382 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:21.122345    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f15feeeb-5e97-4e0e-935a-553affd91bb1-maas-api-xks-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:05:21.122382 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:21.122376    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt2wf\" (UniqueName: \"kubernetes.io/projected/f15feeeb-5e97-4e0e-935a-553affd91bb1-kube-api-access-vt2wf\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:05:21.122382 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:21.122385    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f15feeeb-5e97-4e0e-935a-553affd91bb1-maas-api-openshift-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:05:21.703391 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:21.703356    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a1264-xlgxz" event={"ID":"f15feeeb-5e97-4e0e-935a-553affd91bb1","Type":"ContainerDied","Data":"912ac33966482884fd1fb11fb2730c219d6341879436f5348420c570430c0319"}
Oct 30 17:05:21.703391 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:21.703382    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-873a1264-xlgxz"
Oct 30 17:05:21.703635 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:21.703444    4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="912ac33966482884fd1fb11fb2730c219d6341879436f5348420c570430c0319"
Oct 30 17:05:24.910902 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:24.910864    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:05:27.223430 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:05:27.220865    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:05:27.705431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:27.683498    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:27.705431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:27.683928    4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": context deadline exceeded"
Oct 30 17:05:27.782431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:27.781582    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 17:05:28.080488 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:28.079513    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:28.259061 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:28.257839    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm" podUID="bf43bf3d-f126-40a1-a24f-10214a85ceb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:28.338674 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:28.338547    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:28.691867 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:28.689635    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:28.691867 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:28.690043    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:28.691867 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:28.691468    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 17:05:28.910430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:28.887516    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 17:05:28.924368 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:28.890012    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 17:05:28.925833 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:28.925014    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:05:28.966604 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:28.964311    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f19d8be-9l4rp"]
Oct 30 17:05:28.966604 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:28.965181    4433 cpu_manager.go:414] "containerMap: removing container" podUID="f15feeeb-5e97-4e0e-935a-553affd91bb1" containerName="revoke-keys"
Oct 30 17:05:28.966604 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:28.965201    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f15feeeb-5e97-4e0e-935a-553affd91bb1" containerName="revoke-keys"
Oct 30 17:05:28.966604 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:28.965345    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="f15feeeb-5e97-4e0e-935a-553affd91bb1" containerName="revoke-keys"
Oct 30 17:05:29.043431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:29.043373    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" podUID="458eba2e-eb03-452a-bcf4-a5e91da83acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:29.044188 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:29.044156    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:29.105709 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:29.045031    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f19d8be-9l4rp"
Oct 30 17:05:29.105709 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:29.088521    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:29.105709 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:29.088557    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 30 17:05:29.105709 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:29.104016    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 17:05:29.125917 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:29.125700    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:05:29.125917 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:29.125859    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\""
Oct 30 17:05:29.195561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:29.177314    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f87cc63-37bf-4f29-8605-7c4770c82011-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f19d8be-9l4rp\" (UID: \"7f87cc63-37bf-4f29-8605-7c4770c82011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f19d8be-9l4rp"
Oct 30 17:05:29.242272 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:29.242178    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f87cc63-37bf-4f29-8605-7c4770c82011-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f19d8be-9l4rp\" (UID: \"7f87cc63-37bf-4f29-8605-7c4770c82011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f19d8be-9l4rp"
Oct 30 17:05:29.262746 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:29.262715    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzrqg\" (UniqueName: \"kubernetes.io/projected/7f87cc63-37bf-4f29-8605-7c4770c82011-kube-api-access-kzrqg\") pod \"maas-api-revoke-sub-4f19d8be-9l4rp\" (UID: \"7f87cc63-37bf-4f29-8605-7c4770c82011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f19d8be-9l4rp"
Oct 30 17:05:29.289134 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:29.262447    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm" podUID="bf43bf3d-f126-40a1-a24f-10214a85ceb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:29.324429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:29.299881    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 17:05:29.400558 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:29.371447    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 17:05:29.416752 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:29.416721    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:05:29.464429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:29.425463    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f87cc63-37bf-4f29-8605-7c4770c82011-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f19d8be-9l4rp\" (UID: \"7f87cc63-37bf-4f29-8605-7c4770c82011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f19d8be-9l4rp"
Oct 30 17:05:29.464429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:29.461616    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f87cc63-37bf-4f29-8605-7c4770c82011-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f19d8be-9l4rp\" (UID: \"7f87cc63-37bf-4f29-8605-7c4770c82011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f19d8be-9l4rp"
Oct 30 17:05:29.464429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:29.461791    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzrqg\" (UniqueName: \"kubernetes.io/projected/7f87cc63-37bf-4f29-8605-7c4770c82011-kube-api-access-kzrqg\") pod \"maas-api-revoke-sub-4f19d8be-9l4rp\" (UID: \"7f87cc63-37bf-4f29-8605-7c4770c82011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f19d8be-9l4rp"
Oct 30 17:05:29.464429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:29.463160    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f87cc63-37bf-4f29-8605-7c4770c82011-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f19d8be-9l4rp\" (UID: \"7f87cc63-37bf-4f29-8605-7c4770c82011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f19d8be-9l4rp"
Oct 30 17:05:29.993430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:29.986867    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 17:05:29.999434 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:29.993739    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f87cc63-37bf-4f29-8605-7c4770c82011-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f19d8be-9l4rp\" (UID: \"7f87cc63-37bf-4f29-8605-7c4770c82011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f19d8be-9l4rp"
Oct 30 17:05:30.076593 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:30.068983    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" podUID="458eba2e-eb03-452a-bcf4-a5e91da83acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:30.157678 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:30.104610    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 30 17:05:30.157678 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:30.111687    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:30.157678 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:30.121992    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 17:05:30.157678 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:30.154175    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzrqg\" (UniqueName: \"kubernetes.io/projected/7f87cc63-37bf-4f29-8605-7c4770c82011-kube-api-access-kzrqg\") pod \"maas-api-revoke-sub-4f19d8be-9l4rp\" (UID: \"7f87cc63-37bf-4f29-8605-7c4770c82011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f19d8be-9l4rp"
Oct 30 17:05:30.230451 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:30.227159    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:05:30.234429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:30.231100    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 17:05:30.234720 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:30.234701    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 17:05:30.259715 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:30.259622    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f19d8be-9l4rp"]
Oct 30 17:05:30.318448 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:30.301951    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f19d8be-9l4rp"
Oct 30 17:05:30.318448 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:30.302618    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm" podUID="bf43bf3d-f126-40a1-a24f-10214a85ceb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 30 17:05:30.490432 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:30.450491    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 17:05:30.490432 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:30.483580    4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:30.707435 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:30.701499    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:30.757116 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:30.753382    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 17:05:30.893486 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:30.885055    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 17:05:30.954958 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:30.953520    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:31.001430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:31.000148    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:31.087919 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:31.087872    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" podUID="458eba2e-eb03-452a-bcf4-a5e91da83acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:31.176146 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:31.174385    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:31.362591 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:31.361490    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm" podUID="bf43bf3d-f126-40a1-a24f-10214a85ceb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:31.395432 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:31.393124    4433 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:05:31.395432 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:31.393200    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479" podUID="20dd24a1-9e9c-40d2-8ad8-bf0a206f292e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:31.569487 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:31.564457    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:05:31.620511 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:31.615168    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:31.712312 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:31.712274    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:31.885848 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:31.885745    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:05:31.936707 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:31.936672    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:05:31.936997 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:31.936980    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:05:31.937626 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:31.937595    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:05:31.977729 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:31.966494    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f19d8be-9l4rp"]
Oct 30 17:05:32.000985 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:32.000825    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 17:05:32.212436 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:32.211721    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 17:05:32.222037 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:32.213424    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 17:05:32.223485 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:32.223393    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:05:32.242885 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:32.242857    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 17:05:32.480993 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:32.480956    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f19d8be-9l4rp" event={"ID":"7f87cc63-37bf-4f29-8605-7c4770c82011","Type":"ContainerStarted","Data":"c18a6d37c593ab9888b0e5083b8caaa0bc79c76d3ec9f149eb256e427e8dc16c"}
Oct 30 17:05:33.490488 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:33.490446    4433 generic.go:356] "Generic (PLEG): container finished" podID="7f87cc63-37bf-4f29-8605-7c4770c82011" containerID="1e49f5c71b1694e6eba656d5d36485cce67e267b3640a5c77b8b8c2e4099bb49" exitCode=0
Oct 30 17:05:33.490903 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:33.490522    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f19d8be-9l4rp" event={"ID":"7f87cc63-37bf-4f29-8605-7c4770c82011","Type":"ContainerDied","Data":"1e49f5c71b1694e6eba656d5d36485cce67e267b3640a5c77b8b8c2e4099bb49"}
Oct 30 17:05:34.652792 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:34.652762    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f19d8be-9l4rp"
Oct 30 17:05:34.753025 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:34.752943    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f87cc63-37bf-4f29-8605-7c4770c82011-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f87cc63-37bf-4f29-8605-7c4770c82011-maas-api-openshift-ca\") pod \"7f87cc63-37bf-4f29-8605-7c4770c82011\" (UID: \"7f87cc63-37bf-4f29-8605-7c4770c82011\") "
Oct 30 17:05:34.753025 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:34.752990    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f87cc63-37bf-4f29-8605-7c4770c82011-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f87cc63-37bf-4f29-8605-7c4770c82011-maas-api-xks-ca\") pod \"7f87cc63-37bf-4f29-8605-7c4770c82011\" (UID: \"7f87cc63-37bf-4f29-8605-7c4770c82011\") "
Oct 30 17:05:34.753220 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:34.753168    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f87cc63-37bf-4f29-8605-7c4770c82011-kube-api-access-kzrqg\" (UniqueName: \"kubernetes.io/projected/7f87cc63-37bf-4f29-8605-7c4770c82011-kube-api-access-kzrqg\") pod \"7f87cc63-37bf-4f29-8605-7c4770c82011\" (UID: \"7f87cc63-37bf-4f29-8605-7c4770c82011\") "
Oct 30 17:05:34.753386 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:34.753357    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f87cc63-37bf-4f29-8605-7c4770c82011-maas-api-openshift-ca" pod "7f87cc63-37bf-4f29-8605-7c4770c82011" (UID: "7f87cc63-37bf-4f29-8605-7c4770c82011"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:05:34.753767 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:34.753736    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f87cc63-37bf-4f29-8605-7c4770c82011-maas-api-openshift-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:05:34.755150 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:34.755129    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f87cc63-37bf-4f29-8605-7c4770c82011-maas-api-xks-ca" pod "7f87cc63-37bf-4f29-8605-7c4770c82011" (UID: "7f87cc63-37bf-4f29-8605-7c4770c82011"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:05:34.755323 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:34.755306    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f87cc63-37bf-4f29-8605-7c4770c82011-kube-api-access-kzrqg" pod "7f87cc63-37bf-4f29-8605-7c4770c82011" (UID: "7f87cc63-37bf-4f29-8605-7c4770c82011"). InnerVolumeSpecName "kube-api-access-kzrqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:05:34.855260 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:34.855228    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f87cc63-37bf-4f29-8605-7c4770c82011-maas-api-xks-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:05:34.855260 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:34.855257    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzrqg\" (UniqueName: \"kubernetes.io/projected/7f87cc63-37bf-4f29-8605-7c4770c82011-kube-api-access-kzrqg\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:05:35.505142 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:35.505103    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f19d8be-9l4rp" event={"ID":"7f87cc63-37bf-4f29-8605-7c4770c82011","Type":"ContainerDied","Data":"c18a6d37c593ab9888b0e5083b8caaa0bc79c76d3ec9f149eb256e427e8dc16c"}
Oct 30 17:05:35.505324 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:35.505162    4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c18a6d37c593ab9888b0e5083b8caaa0bc79c76d3ec9f149eb256e427e8dc16c"
Oct 30 17:05:35.505324 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:35.505115    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f19d8be-9l4rp"
Oct 30 17:05:37.736897 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:37.736858    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"]
Oct 30 17:05:37.737674 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:37.737653    4433 cpu_manager.go:414] "containerMap: removing container" podUID="7f87cc63-37bf-4f29-8605-7c4770c82011" containerName="revoke-keys"
Oct 30 17:05:37.737728 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:37.737676    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f87cc63-37bf-4f29-8605-7c4770c82011" containerName="revoke-keys"
Oct 30 17:05:37.737781 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:37.737769    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f87cc63-37bf-4f29-8605-7c4770c82011" containerName="revoke-keys"
Oct 30 17:05:37.741889 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:37.741872    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"
Oct 30 17:05:37.773189 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:37.773156    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"]
Oct 30 17:05:37.796309 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:37.796270    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/946311cc-f655-4798-bce5-10ad7d73c9e6-workload-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"
Oct 30 17:05:37.796586 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:37.796555    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/946311cc-f655-4798-bce5-10ad7d73c9e6-istio-data\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"
Oct 30 17:05:37.796696 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:37.796672    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4sdd\" (UniqueName: \"kubernetes.io/projected/946311cc-f655-4798-bce5-10ad7d73c9e6-kube-api-access-n4sdd\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"
Oct 30 17:05:37.796847 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:37.796812    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/946311cc-f655-4798-bce5-10ad7d73c9e6-istio-podinfo\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"
Oct 30 17:05:37.796962 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:37.796945    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/946311cc-f655-4798-bce5-10ad7d73c9e6-istiod-ca-cert\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"
Oct 30 17:05:37.797012 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:37.796984    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/946311cc-f655-4798-bce5-10ad7d73c9e6-workload-certs\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"
Oct 30 17:05:37.797093 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:37.797072    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/946311cc-f655-4798-bce5-10ad7d73c9e6-credential-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"
Oct 30 17:05:37.797137 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:37.797106    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/946311cc-f655-4798-bce5-10ad7d73c9e6-istio-envoy\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"
Oct 30 17:05:37.797380 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:37.797362    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/946311cc-f655-4798-bce5-10ad7d73c9e6-istio-token\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"
Oct 30 17:05:37.898320 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:37.898281    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4sdd\" (UniqueName: \"kubernetes.io/projected/946311cc-f655-4798-bce5-10ad7d73c9e6-kube-api-access-n4sdd\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"
Oct 30 17:05:37.898547 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:37.898359    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/946311cc-f655-4798-bce5-10ad7d73c9e6-istio-podinfo\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"
Oct 30 17:05:37.898547 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:37.898452    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/946311cc-f655-4798-bce5-10ad7d73c9e6-istiod-ca-cert\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"
Oct 30 17:05:37.898547 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:37.898490    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/946311cc-f655-4798-bce5-10ad7d73c9e6-workload-certs\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"
Oct 30 17:05:37.898547 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:37.898521    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/946311cc-f655-4798-bce5-10ad7d73c9e6-credential-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"
Oct 30 17:05:37.898547 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:37.898545    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/946311cc-f655-4798-bce5-10ad7d73c9e6-istio-envoy\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"
Oct 30 17:05:37.898822 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:37.898701    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/946311cc-f655-4798-bce5-10ad7d73c9e6-istio-token\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"
Oct 30 17:05:37.898822 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:37.898759    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/946311cc-f655-4798-bce5-10ad7d73c9e6-workload-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"
Oct 30 17:05:37.898952 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:37.898929    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/946311cc-f655-4798-bce5-10ad7d73c9e6-workload-certs\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"
Oct 30 17:05:37.899005 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:37.898982    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/946311cc-f655-4798-bce5-10ad7d73c9e6-credential-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"
Oct 30 17:05:37.899045 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:37.898986    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/946311cc-f655-4798-bce5-10ad7d73c9e6-istio-data\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"
Oct 30 17:05:37.899293 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:37.899269    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/946311cc-f655-4798-bce5-10ad7d73c9e6-workload-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"
Oct 30 17:05:37.899394 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:37.899291    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/946311cc-f655-4798-bce5-10ad7d73c9e6-istiod-ca-cert\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"
Oct 30 17:05:37.899394 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:37.899368    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/946311cc-f655-4798-bce5-10ad7d73c9e6-istio-data\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"
Oct 30 17:05:37.901151 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:37.901127    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/946311cc-f655-4798-bce5-10ad7d73c9e6-istio-podinfo\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"
Oct 30 17:05:37.901151 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:37.901142    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/946311cc-f655-4798-bce5-10ad7d73c9e6-istio-envoy\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"
Oct 30 17:05:37.932983 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:37.932952    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/946311cc-f655-4798-bce5-10ad7d73c9e6-istio-token\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"
Oct 30 17:05:37.933154 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:37.933130    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4sdd\" (UniqueName: \"kubernetes.io/projected/946311cc-f655-4798-bce5-10ad7d73c9e6-kube-api-access-n4sdd\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"
Oct 30 17:05:38.057526 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.057395    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"
Oct 30 17:05:38.192124 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.192090    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"]
Oct 30 17:05:38.197906 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.197883    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"
Oct 30 17:05:38.219639 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.219611    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"]
Oct 30 17:05:38.305213 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.305178    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a82e355-1770-48e7-a8ab-92d5d1b6d257-workload-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"
Oct 30 17:05:38.305376 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.305221    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a82e355-1770-48e7-a8ab-92d5d1b6d257-istio-data\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"
Oct 30 17:05:38.305376 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.305337    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a82e355-1770-48e7-a8ab-92d5d1b6d257-istiod-ca-cert\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"
Oct 30 17:05:38.305376 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.305370    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a82e355-1770-48e7-a8ab-92d5d1b6d257-credential-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"
Oct 30 17:05:38.305530 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.305510    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a82e355-1770-48e7-a8ab-92d5d1b6d257-workload-certs\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"
Oct 30 17:05:38.305570 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.305544    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a82e355-1770-48e7-a8ab-92d5d1b6d257-istio-envoy\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"
Oct 30 17:05:38.305570 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.305567    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a82e355-1770-48e7-a8ab-92d5d1b6d257-istio-token\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"
Oct 30 17:05:38.305660 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.305645    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a82e355-1770-48e7-a8ab-92d5d1b6d257-istio-podinfo\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"
Oct 30 17:05:38.305696 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.305678    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj6rj\" (UniqueName: \"kubernetes.io/projected/1a82e355-1770-48e7-a8ab-92d5d1b6d257-kube-api-access-tj6rj\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"
Oct 30 17:05:38.377728 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.377697    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"]
Oct 30 17:05:38.378898 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:05:38.378873    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946311cc_f655_4798_bce5_10ad7d73c9e6.slice/crio-434fbe079508e89f7752fc54cf6937d29730bfd988f66d85382c9d19d83efab0 WatchSource:0}: Error finding container 434fbe079508e89f7752fc54cf6937d29730bfd988f66d85382c9d19d83efab0: Status 404 returned error can't find the container with id 434fbe079508e89f7752fc54cf6937d29730bfd988f66d85382c9d19d83efab0
Oct 30 17:05:38.381377 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.381347    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:05:38.381492 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.381429    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:05:38.381492 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.381467    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:05:38.407588 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.407561    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a82e355-1770-48e7-a8ab-92d5d1b6d257-workload-certs\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"
Oct 30 17:05:38.407736 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.407606    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a82e355-1770-48e7-a8ab-92d5d1b6d257-istio-envoy\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"
Oct 30 17:05:38.407736 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.407632    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a82e355-1770-48e7-a8ab-92d5d1b6d257-istio-token\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"
Oct 30 17:05:38.407736 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.407683    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a82e355-1770-48e7-a8ab-92d5d1b6d257-istio-podinfo\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"
Oct 30 17:05:38.407736 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.407707    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj6rj\" (UniqueName: \"kubernetes.io/projected/1a82e355-1770-48e7-a8ab-92d5d1b6d257-kube-api-access-tj6rj\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"
Oct 30 17:05:38.407993 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.407845    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a82e355-1770-48e7-a8ab-92d5d1b6d257-workload-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"
Oct 30 17:05:38.407993 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.407880    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a82e355-1770-48e7-a8ab-92d5d1b6d257-istio-data\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"
Oct 30 17:05:38.407993 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.407946    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a82e355-1770-48e7-a8ab-92d5d1b6d257-istiod-ca-cert\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"
Oct 30 17:05:38.407993 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.407974    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a82e355-1770-48e7-a8ab-92d5d1b6d257-credential-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"
Oct 30 17:05:38.408195 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.408029    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a82e355-1770-48e7-a8ab-92d5d1b6d257-workload-certs\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"
Oct 30 17:05:38.408299 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.408276    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a82e355-1770-48e7-a8ab-92d5d1b6d257-istio-data\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"
Oct 30 17:05:38.408362 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.408332    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a82e355-1770-48e7-a8ab-92d5d1b6d257-credential-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"
Oct 30 17:05:38.408511 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.408486    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a82e355-1770-48e7-a8ab-92d5d1b6d257-workload-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"
Oct 30 17:05:38.408712 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.408693    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a82e355-1770-48e7-a8ab-92d5d1b6d257-istiod-ca-cert\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"
Oct 30 17:05:38.410164 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.410143    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a82e355-1770-48e7-a8ab-92d5d1b6d257-istio-envoy\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"
Oct 30 17:05:38.410269 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.410253    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a82e355-1770-48e7-a8ab-92d5d1b6d257-istio-podinfo\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"
Oct 30 17:05:38.430775 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.430745    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a82e355-1770-48e7-a8ab-92d5d1b6d257-istio-token\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"
Oct 30 17:05:38.430920 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.430902    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj6rj\" (UniqueName: \"kubernetes.io/projected/1a82e355-1770-48e7-a8ab-92d5d1b6d257-kube-api-access-tj6rj\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"
Oct 30 17:05:38.513074 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.513030    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"
Oct 30 17:05:38.528840 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.528458    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp" event={"ID":"946311cc-f655-4798-bce5-10ad7d73c9e6","Type":"ContainerStarted","Data":"f2cbcd6bafafd3e966f436bf1923a9997029bcb4c93a7058cc01ee3e463ab3b4"}
Oct 30 17:05:38.528840 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.528523    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp" event={"ID":"946311cc-f655-4798-bce5-10ad7d73c9e6","Type":"ContainerStarted","Data":"434fbe079508e89f7752fc54cf6937d29730bfd988f66d85382c9d19d83efab0"}
Oct 30 17:05:38.578270 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.578220    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp" podStartSLOduration=1.578203848 podStartE2EDuration="1.578203848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:38.575262588 +0000 UTC m=+1115.010164786" watchObservedRunningTime="2026-10-30 17:05:38.578203848 +0000 UTC m=+1115.013106053"
Oct 30 17:05:38.843981 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.843948    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"]
Oct 30 17:05:38.845005 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:05:38.844975    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a82e355_1770_48e7_a8ab_92d5d1b6d257.slice/crio-cdcb8d9e44eab83b54ecca8a28d3071de80a63fc8f09f4a4f66bf04f928dde08 WatchSource:0}: Error finding container cdcb8d9e44eab83b54ecca8a28d3071de80a63fc8f09f4a4f66bf04f928dde08: Status 404 returned error can't find the container with id cdcb8d9e44eab83b54ecca8a28d3071de80a63fc8f09f4a4f66bf04f928dde08
Oct 30 17:05:38.847732 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.847699    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:05:38.847857 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.847777    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:05:38.847857 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:38.847831    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:05:39.060815 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:39.060630    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"
Oct 30 17:05:39.062659 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:39.062578    4433 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 17:05:39.063186 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:39.063090    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp" podUID="946311cc-f655-4798-bce5-10ad7d73c9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 17:05:39.545168 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:39.545125    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw" event={"ID":"1a82e355-1770-48e7-a8ab-92d5d1b6d257","Type":"ContainerStarted","Data":"adc5cfb160ede6a8214cee4f1864243b93292e6761062e1809b31e6367de7ed2"}
Oct 30 17:05:39.545368 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:39.545181    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw" event={"ID":"1a82e355-1770-48e7-a8ab-92d5d1b6d257","Type":"ContainerStarted","Data":"cdcb8d9e44eab83b54ecca8a28d3071de80a63fc8f09f4a4f66bf04f928dde08"}
Oct 30 17:05:39.641610 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:39.638415    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw" podStartSLOduration=1.638379464 podStartE2EDuration="1.638379464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:39.63310733 +0000 UTC m=+1116.068009536" watchObservedRunningTime="2026-10-30 17:05:39.638379464 +0000 UTC m=+1116.073281675"
Oct 30 17:05:40.070430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:40.058833    4433 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 17:05:40.070430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:40.058897    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp" podUID="946311cc-f655-4798-bce5-10ad7d73c9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 17:05:40.529431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:40.518750    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"
Oct 30 17:05:40.529431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:40.525564    4433 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 17:05:40.529431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:40.525629    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw" podUID="1a82e355-1770-48e7-a8ab-92d5d1b6d257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 17:05:41.063429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:41.059941    4433 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 17:05:41.063429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:41.060016    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp" podUID="946311cc-f655-4798-bce5-10ad7d73c9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 17:05:41.514242 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:41.514206    4433 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 17:05:41.514712 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:41.514259    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw" podUID="1a82e355-1770-48e7-a8ab-92d5d1b6d257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 17:05:42.070433 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:42.063942    4433 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 17:05:42.070433 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:42.064009    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp" podUID="946311cc-f655-4798-bce5-10ad7d73c9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 17:05:42.514216 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:42.514020    4433 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 17:05:42.514216 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:42.514112    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw" podUID="1a82e355-1770-48e7-a8ab-92d5d1b6d257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 17:05:43.078748 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:43.070242    4433 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 17:05:43.078748 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:43.070329    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp" podUID="946311cc-f655-4798-bce5-10ad7d73c9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 17:05:43.513841 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:43.513803    4433 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 17:05:43.514092 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:43.514065    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw" podUID="1a82e355-1770-48e7-a8ab-92d5d1b6d257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 17:05:44.060356 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:44.060321    4433 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 17:05:44.061005 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:44.060954    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp" podUID="946311cc-f655-4798-bce5-10ad7d73c9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 17:05:44.535557 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:44.535520    4433 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 17:05:44.535898 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:44.535844    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw" podUID="1a82e355-1770-48e7-a8ab-92d5d1b6d257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 17:05:45.060439 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:45.059821    4433 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 17:05:45.060439 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:45.059883    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp" podUID="946311cc-f655-4798-bce5-10ad7d73c9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 17:05:45.514149 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:45.514114    4433 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 17:05:45.514349 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:45.514177    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw" podUID="1a82e355-1770-48e7-a8ab-92d5d1b6d257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 17:05:46.063794 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:46.063755    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"
Oct 30 17:05:46.064297 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:46.064282    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"
Oct 30 17:05:46.065066 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:46.065039    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"
Oct 30 17:05:46.525279 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:46.524511    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"
Oct 30 17:05:46.525559 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:46.525530    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"
Oct 30 17:05:46.527066 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:46.527038    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"
Oct 30 17:05:53.478906 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:53.478871    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"]
Oct 30 17:05:53.483484 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:53.483457    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"
Oct 30 17:05:53.487377 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:53.487347    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be1ee2aa-dabc-4fab-b1d6-24c01758a780-istio-data\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"
Oct 30 17:05:53.487576 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:53.487554    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be1ee2aa-dabc-4fab-b1d6-24c01758a780-istiod-ca-cert\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"
Oct 30 17:05:53.487656 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:53.487592    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be1ee2aa-dabc-4fab-b1d6-24c01758a780-workload-certs\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"
Oct 30 17:05:53.487656 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:53.487622    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be1ee2aa-dabc-4fab-b1d6-24c01758a780-istio-podinfo\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"
Oct 30 17:05:53.487809 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:53.487784    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgv7j\" (UniqueName: \"kubernetes.io/projected/be1ee2aa-dabc-4fab-b1d6-24c01758a780-kube-api-access-sgv7j\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"
Oct 30 17:05:53.487899 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:53.487846    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1ee2aa-dabc-4fab-b1d6-24c01758a780-credential-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"
Oct 30 17:05:53.487937 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:53.487896    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be1ee2aa-dabc-4fab-b1d6-24c01758a780-istio-envoy\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"
Oct 30 17:05:53.488012 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:53.487995    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1ee2aa-dabc-4fab-b1d6-24c01758a780-workload-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"
Oct 30 17:05:53.488077 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:53.488051    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be1ee2aa-dabc-4fab-b1d6-24c01758a780-istio-token\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"
Oct 30 17:05:53.506940 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:53.506908    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"]
Oct 30 17:05:53.589553 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:53.589515    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be1ee2aa-dabc-4fab-b1d6-24c01758a780-istio-envoy\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"
Oct 30 17:05:53.589728 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:53.589630    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1ee2aa-dabc-4fab-b1d6-24c01758a780-workload-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"
Oct 30 17:05:53.589728 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:53.589684    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be1ee2aa-dabc-4fab-b1d6-24c01758a780-istio-token\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"
Oct 30 17:05:53.589825 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:53.589808    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be1ee2aa-dabc-4fab-b1d6-24c01758a780-istio-data\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"
Oct 30 17:05:53.589864 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:53.589835    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be1ee2aa-dabc-4fab-b1d6-24c01758a780-istiod-ca-cert\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"
Oct 30 17:05:53.589918 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:53.589864    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be1ee2aa-dabc-4fab-b1d6-24c01758a780-workload-certs\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"
Oct 30 17:05:53.589918 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:53.589897    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be1ee2aa-dabc-4fab-b1d6-24c01758a780-istio-podinfo\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"
Oct 30 17:05:53.590028 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:53.590007    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgv7j\" (UniqueName: \"kubernetes.io/projected/be1ee2aa-dabc-4fab-b1d6-24c01758a780-kube-api-access-sgv7j\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"
Oct 30 17:05:53.590083 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:53.590062    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1ee2aa-dabc-4fab-b1d6-24c01758a780-credential-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"
Oct 30 17:05:53.590144 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:53.590072    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1ee2aa-dabc-4fab-b1d6-24c01758a780-workload-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"
Oct 30 17:05:53.590258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:53.590190    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be1ee2aa-dabc-4fab-b1d6-24c01758a780-istio-data\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"
Oct 30 17:05:53.590431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:53.590238    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be1ee2aa-dabc-4fab-b1d6-24c01758a780-workload-certs\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"
Oct 30 17:05:53.590623 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:53.590604    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1ee2aa-dabc-4fab-b1d6-24c01758a780-credential-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"
Oct 30 17:05:53.590686 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:53.590665    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be1ee2aa-dabc-4fab-b1d6-24c01758a780-istiod-ca-cert\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"
Oct 30 17:05:53.592154 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:53.592131    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be1ee2aa-dabc-4fab-b1d6-24c01758a780-istio-envoy\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"
Oct 30 17:05:53.592728 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:53.592707    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be1ee2aa-dabc-4fab-b1d6-24c01758a780-istio-podinfo\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"
Oct 30 17:05:53.613955 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:53.613921    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be1ee2aa-dabc-4fab-b1d6-24c01758a780-istio-token\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"
Oct 30 17:05:53.614109 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:53.614038    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgv7j\" (UniqueName: \"kubernetes.io/projected/be1ee2aa-dabc-4fab-b1d6-24c01758a780-kube-api-access-sgv7j\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"
Oct 30 17:05:53.799008 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:53.798922    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"
Oct 30 17:05:54.124618 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:54.124590    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"]
Oct 30 17:05:54.125372 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:05:54.125347    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe1ee2aa_dabc_4fab_b1d6_24c01758a780.slice/crio-20ccb2a3f431f39606336bc12c543c6441e8ee4df5caeebe508c04959ec3f528 WatchSource:0}: Error finding container 20ccb2a3f431f39606336bc12c543c6441e8ee4df5caeebe508c04959ec3f528: Status 404 returned error can't find the container with id 20ccb2a3f431f39606336bc12c543c6441e8ee4df5caeebe508c04959ec3f528
Oct 30 17:05:54.128239 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:54.128205    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:05:54.128322 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:54.128274    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:05:54.128322 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:54.128304    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:05:54.742233 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:54.742196    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf" event={"ID":"be1ee2aa-dabc-4fab-b1d6-24c01758a780","Type":"ContainerStarted","Data":"96c9190092fb55255a27ea764d965acbb55a32c6598f1227ddb222d12784f128"}
Oct 30 17:05:54.742703 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:54.742251    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf" event={"ID":"be1ee2aa-dabc-4fab-b1d6-24c01758a780","Type":"ContainerStarted","Data":"20ccb2a3f431f39606336bc12c543c6441e8ee4df5caeebe508c04959ec3f528"}
Oct 30 17:05:54.799865 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:54.799827    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"
Oct 30 17:05:54.813370 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:54.813317    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf" podStartSLOduration=1.81330299 podStartE2EDuration="1.81330299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:54.798575981 +0000 UTC m=+1131.233478183" watchObservedRunningTime="2026-10-30 17:05:54.81330299 +0000 UTC m=+1131.248205258"
Oct 30 17:05:55.801595 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:55.801556    4433 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 17:05:55.802134 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:55.801611    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf" podUID="be1ee2aa-dabc-4fab-b1d6-24c01758a780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 17:05:56.800134 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:56.800101    4433 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 17:05:56.800323 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:56.800160    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf" podUID="be1ee2aa-dabc-4fab-b1d6-24c01758a780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 17:05:57.004444 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:05:57.004391    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:05:57.799970 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:57.799932    4433 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 17:05:57.800153 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:57.799991    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf" podUID="be1ee2aa-dabc-4fab-b1d6-24c01758a780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 17:05:58.800244 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:58.800207    4433 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 17:05:58.800668 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:58.800263    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf" podUID="be1ee2aa-dabc-4fab-b1d6-24c01758a780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 17:05:59.799491 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:59.799458    4433 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 17:05:59.799661 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:05:59.799508    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf" podUID="be1ee2aa-dabc-4fab-b1d6-24c01758a780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 17:06:00.799280 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:00.799244    4433 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 17:06:00.799832 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:00.799301    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf" podUID="be1ee2aa-dabc-4fab-b1d6-24c01758a780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 17:06:01.811855 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:01.811801    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"
Oct 30 17:06:01.812821 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:01.812799    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"
Oct 30 17:06:01.814852 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:01.814829    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"
Oct 30 17:06:04.730430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:04.729942    4433 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:06:04.730430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:04.730029    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz" podUID="e6431be3-b51d-40bc-bc87-ddf189905e14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:06:05.142559 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:05.128731    4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:06:05.384855 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:05.384797    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm" podUID="bf43bf3d-f126-40a1-a24f-10214a85ceb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 30 17:06:07.719834 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:07.719791    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w"]
Oct 30 17:06:07.724944 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:07.724923    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w"
Oct 30 17:06:07.766184 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:07.766149    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-1cdec0e0-serving-cert\""
Oct 30 17:06:07.773496 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:07.772105    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w"]
Oct 30 17:06:07.883333 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:07.883297    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq5zx\" (UniqueName: \"kubernetes.io/projected/5d3e3542-3d7f-483d-9c05-49e437e9a930-kube-api-access-dq5zx\") pod \"maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w\" (UID: \"5d3e3542-3d7f-483d-9c05-49e437e9a930\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w"
Oct 30 17:06:07.883525 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:07.883492    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3e3542-3d7f-483d-9c05-49e437e9a930-openshift-service-ca\") pod \"maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w\" (UID: \"5d3e3542-3d7f-483d-9c05-49e437e9a930\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w"
Oct 30 17:06:07.883610 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:07.883596    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d3e3542-3d7f-483d-9c05-49e437e9a930-metrics-certs\") pod \"maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w\" (UID: \"5d3e3542-3d7f-483d-9c05-49e437e9a930\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w"
Oct 30 17:06:07.883661 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:07.883624    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d3e3542-3d7f-483d-9c05-49e437e9a930-maas-api-tls\") pod \"maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w\" (UID: \"5d3e3542-3d7f-483d-9c05-49e437e9a930\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w"
Oct 30 17:06:07.985314 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:07.985227    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3e3542-3d7f-483d-9c05-49e437e9a930-openshift-service-ca\") pod \"maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w\" (UID: \"5d3e3542-3d7f-483d-9c05-49e437e9a930\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w"
Oct 30 17:06:07.985548 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:07.985346    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d3e3542-3d7f-483d-9c05-49e437e9a930-metrics-certs\") pod \"maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w\" (UID: \"5d3e3542-3d7f-483d-9c05-49e437e9a930\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w"
Oct 30 17:06:07.985548 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:07.985380    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d3e3542-3d7f-483d-9c05-49e437e9a930-maas-api-tls\") pod \"maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w\" (UID: \"5d3e3542-3d7f-483d-9c05-49e437e9a930\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w"
Oct 30 17:06:07.985548 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:07.985526    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq5zx\" (UniqueName: \"kubernetes.io/projected/5d3e3542-3d7f-483d-9c05-49e437e9a930-kube-api-access-dq5zx\") pod \"maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w\" (UID: \"5d3e3542-3d7f-483d-9c05-49e437e9a930\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w"
Oct 30 17:06:07.986035 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:07.986008    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3e3542-3d7f-483d-9c05-49e437e9a930-openshift-service-ca\") pod \"maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w\" (UID: \"5d3e3542-3d7f-483d-9c05-49e437e9a930\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w"
Oct 30 17:06:07.988027 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:07.988005    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d3e3542-3d7f-483d-9c05-49e437e9a930-maas-api-tls\") pod \"maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w\" (UID: \"5d3e3542-3d7f-483d-9c05-49e437e9a930\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w"
Oct 30 17:06:07.988110 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:07.988044    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d3e3542-3d7f-483d-9c05-49e437e9a930-metrics-certs\") pod \"maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w\" (UID: \"5d3e3542-3d7f-483d-9c05-49e437e9a930\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w"
Oct 30 17:06:08.008953 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:08.008917    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq5zx\" (UniqueName: \"kubernetes.io/projected/5d3e3542-3d7f-483d-9c05-49e437e9a930-kube-api-access-dq5zx\") pod \"maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w\" (UID: \"5d3e3542-3d7f-483d-9c05-49e437e9a930\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w"
Oct 30 17:06:08.039917 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:08.039886    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w"
Oct 30 17:06:08.351955 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:08.351926    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w"]
Oct 30 17:06:08.354300 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:06:08.354265    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d3e3542_3d7f_483d_9c05_49e437e9a930.slice/crio-79479fe38c8ece653f49d08d8ff0f599093bd92325a878b75f6605b1dfc94e0d WatchSource:0}: Error finding container 79479fe38c8ece653f49d08d8ff0f599093bd92325a878b75f6605b1dfc94e0d: Status 404 returned error can't find the container with id 79479fe38c8ece653f49d08d8ff0f599093bd92325a878b75f6605b1dfc94e0d
Oct 30 17:06:08.356048 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:08.356025    4433 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 17:06:08.965933 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:08.965899    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w" event={"ID":"5d3e3542-3d7f-483d-9c05-49e437e9a930","Type":"ContainerStarted","Data":"79479fe38c8ece653f49d08d8ff0f599093bd92325a878b75f6605b1dfc94e0d"}
Oct 30 17:06:09.975114 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:09.975079    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w" event={"ID":"5d3e3542-3d7f-483d-9c05-49e437e9a930","Type":"ContainerStarted","Data":"ee2d8303888e2a390263ba26d1edf32539ff800009f16a5563539d439deee926"}
Oct 30 17:06:09.975621 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:09.975200    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w"
Oct 30 17:06:10.016568 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:10.016511    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w" podStartSLOduration=2.485298431 podStartE2EDuration="3.016496071s" totalImagesPullingTime="531.19764ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:06:10.015677809 +0000 UTC m=+1146.450580015" watchObservedRunningTime="2026-10-30 17:06:10.016496071 +0000 UTC m=+1146.451398268"
Oct 30 17:06:15.987636 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:15.987604    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w"
Oct 30 17:06:16.032640 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:16.032602    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e9671d-xmwqb"]
Oct 30 17:06:16.050863 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:16.050775    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e9671d-xmwqb"]
Oct 30 17:06:17.697246 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:17.697210    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e098fdc-ae03-4286-9891-cd5a81343358" path="/var/lib/kubelet/pods/3e098fdc-ae03-4286-9891-cd5a81343358/volumes"
Oct 30 17:06:19.031951 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:19.031916    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1cfa206-5m555"]
Oct 30 17:06:19.043716 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:19.043681    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b089650-8x8fb"]
Oct 30 17:06:19.047398 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:19.047372    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1cfa206-5m555"]
Oct 30 17:06:19.051041 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:19.051016    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b089650-8x8fb"]
Oct 30 17:06:19.699937 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:19.699892    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08dd97a0-74fd-4b66-84ff-47717c5b8849" path="/var/lib/kubelet/pods/08dd97a0-74fd-4b66-84ff-47717c5b8849/volumes"
Oct 30 17:06:19.700568 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:19.700534    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8edded39-42a8-4d0b-bfb1-d5b77d3102fd" path="/var/lib/kubelet/pods/8edded39-42a8-4d0b-bfb1-d5b77d3102fd/volumes"
Oct 30 17:06:23.036946 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:23.036883    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24756d7a-pwrs7"]
Oct 30 17:06:23.047950 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:23.047917    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24756d7a-pwrs7"]
Oct 30 17:06:23.572039 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:23.571994    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"]
Oct 30 17:06:23.576506 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:23.576482    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"
Oct 30 17:06:23.628649 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:23.628608    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea8fea41-d6c6-414b-8a0d-770c435d514c-credential-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"
Oct 30 17:06:23.628848 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:23.628670    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea8fea41-d6c6-414b-8a0d-770c435d514c-istio-data\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"
Oct 30 17:06:23.628848 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:23.628752    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea8fea41-d6c6-414b-8a0d-770c435d514c-istio-token\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"
Oct 30 17:06:23.641771 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:23.639031    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea8fea41-d6c6-414b-8a0d-770c435d514c-istio-podinfo\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"
Oct 30 17:06:23.641771 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:23.639103    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea8fea41-d6c6-414b-8a0d-770c435d514c-istiod-ca-cert\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"
Oct 30 17:06:23.641771 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:23.639315    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea8fea41-d6c6-414b-8a0d-770c435d514c-workload-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"
Oct 30 17:06:23.641771 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:23.639607    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea8fea41-d6c6-414b-8a0d-770c435d514c-istio-envoy\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"
Oct 30 17:06:23.641771 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:23.639647    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7svfc\" (UniqueName: \"kubernetes.io/projected/ea8fea41-d6c6-414b-8a0d-770c435d514c-kube-api-access-7svfc\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"
Oct 30 17:06:23.641771 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:23.639742    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea8fea41-d6c6-414b-8a0d-770c435d514c-workload-certs\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"
Oct 30 17:06:23.666300 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:23.666261    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"]
Oct 30 17:06:23.702695 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:23.702662    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7c0d558-7283-4304-a4a6-1a9a110ba265" path="/var/lib/kubelet/pods/d7c0d558-7283-4304-a4a6-1a9a110ba265/volumes"
Oct 30 17:06:23.740783 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:23.740747    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea8fea41-d6c6-414b-8a0d-770c435d514c-workload-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"
Oct 30 17:06:23.740963 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:23.740915    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea8fea41-d6c6-414b-8a0d-770c435d514c-istio-envoy\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"
Oct 30 17:06:23.740963 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:23.740944    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7svfc\" (UniqueName: \"kubernetes.io/projected/ea8fea41-d6c6-414b-8a0d-770c435d514c-kube-api-access-7svfc\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"
Oct 30 17:06:23.741043 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:23.740997    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea8fea41-d6c6-414b-8a0d-770c435d514c-workload-certs\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"
Oct 30 17:06:23.741091 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:23.741042    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea8fea41-d6c6-414b-8a0d-770c435d514c-credential-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"
Oct 30 17:06:23.741091 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:23.741070    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea8fea41-d6c6-414b-8a0d-770c435d514c-istio-data\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"
Oct 30 17:06:23.741160 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:23.741102    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea8fea41-d6c6-414b-8a0d-770c435d514c-istio-token\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"
Oct 30 17:06:23.741160 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:23.741117    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea8fea41-d6c6-414b-8a0d-770c435d514c-istio-podinfo\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"
Oct 30 17:06:23.741160 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:23.741142    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea8fea41-d6c6-414b-8a0d-770c435d514c-istiod-ca-cert\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"
Oct 30 17:06:23.741790 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:23.741766    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea8fea41-d6c6-414b-8a0d-770c435d514c-istiod-ca-cert\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"
Oct 30 17:06:23.741981 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:23.741962    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea8fea41-d6c6-414b-8a0d-770c435d514c-workload-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"
Oct 30 17:06:23.742487 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:23.742466    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea8fea41-d6c6-414b-8a0d-770c435d514c-credential-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"
Oct 30 17:06:23.742886 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:23.742866    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea8fea41-d6c6-414b-8a0d-770c435d514c-workload-certs\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"
Oct 30 17:06:23.743291 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:23.743272    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea8fea41-d6c6-414b-8a0d-770c435d514c-istio-data\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"
Oct 30 17:06:23.744633 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:23.744610    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea8fea41-d6c6-414b-8a0d-770c435d514c-istio-envoy\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"
Oct 30 17:06:23.745675 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:23.745652    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea8fea41-d6c6-414b-8a0d-770c435d514c-istio-podinfo\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"
Oct 30 17:06:23.766221 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:23.766186    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea8fea41-d6c6-414b-8a0d-770c435d514c-istio-token\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"
Oct 30 17:06:23.767963 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:23.767938    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7svfc\" (UniqueName: \"kubernetes.io/projected/ea8fea41-d6c6-414b-8a0d-770c435d514c-kube-api-access-7svfc\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"
Oct 30 17:06:23.957645 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:23.957610    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"
Oct 30 17:06:24.282597 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:24.282561    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"]
Oct 30 17:06:24.283761 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:06:24.283733    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea8fea41_d6c6_414b_8a0d_770c435d514c.slice/crio-6bb9d2548a58e8dc5f3fe94ddb1a50b40458afe5558d7770bce608721ed77953 WatchSource:0}: Error finding container 6bb9d2548a58e8dc5f3fe94ddb1a50b40458afe5558d7770bce608721ed77953: Status 404 returned error can't find the container with id 6bb9d2548a58e8dc5f3fe94ddb1a50b40458afe5558d7770bce608721ed77953
Oct 30 17:06:24.286631 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:24.286597    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:06:24.286699 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:24.286666    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:06:24.286735 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:24.286697    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:06:25.107550 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:25.107518    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7" event={"ID":"ea8fea41-d6c6-414b-8a0d-770c435d514c","Type":"ContainerStarted","Data":"7e72b79430fac5e029ffbc3bcf7602b8ebc2b0d7af89cfb018279301a854497d"}
Oct 30 17:06:25.107725 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:25.107569    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7" event={"ID":"ea8fea41-d6c6-414b-8a0d-770c435d514c","Type":"ContainerStarted","Data":"6bb9d2548a58e8dc5f3fe94ddb1a50b40458afe5558d7770bce608721ed77953"}
Oct 30 17:06:25.177088 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:25.177011    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7" podStartSLOduration=2.176991512 podStartE2EDuration="2.176991512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:06:25.176043053 +0000 UTC m=+1161.610945260" watchObservedRunningTime="2026-10-30 17:06:25.176991512 +0000 UTC m=+1161.611893719"
Oct 30 17:06:25.968429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:25.959335    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"
Oct 30 17:06:25.968429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:25.959486    4433 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 17:06:25.968429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:25.959533    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7" podUID="ea8fea41-d6c6-414b-8a0d-770c435d514c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 17:06:26.962427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:26.961764    4433 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 17:06:26.962427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:26.961828    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7" podUID="ea8fea41-d6c6-414b-8a0d-770c435d514c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 17:06:27.003497 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:06:27.003455    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:06:27.959023 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:27.958750    4433 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 17:06:27.959023 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:27.958806    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7" podUID="ea8fea41-d6c6-414b-8a0d-770c435d514c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 17:06:28.960913 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:28.960872    4433 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 17:06:28.966670 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:28.961481    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7" podUID="ea8fea41-d6c6-414b-8a0d-770c435d514c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 17:06:29.972741 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:29.971036    4433 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 17:06:29.972741 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:29.971114    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7" podUID="ea8fea41-d6c6-414b-8a0d-770c435d514c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 17:06:30.959019 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:30.958982    4433 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 17:06:30.959208 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:30.959038    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7" podUID="ea8fea41-d6c6-414b-8a0d-770c435d514c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 17:06:32.130210 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:32.130157    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"
Oct 30 17:06:32.131222 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:32.131075    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"
Oct 30 17:06:32.132269 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:32.132248    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"
Oct 30 17:06:33.033421 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:33.033364    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-233c3d40-4prqw"]
Oct 30 17:06:33.044445 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:33.044412    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-233c3d40-4prqw"]
Oct 30 17:06:33.710309 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:33.710274    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76f81542-3443-4db5-a900-f721f2e05e38" path="/var/lib/kubelet/pods/76f81542-3443-4db5-a900-f721f2e05e38/volumes"
Oct 30 17:06:40.621689 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:40.621652    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:06:43.901588 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:43.901539    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:06:43.951684 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:43.951629    4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:06:44.377438 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:44.375783    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm" podUID="bf43bf3d-f126-40a1-a24f-10214a85ceb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:06:44.546656 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:44.545520    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" podUID="458eba2e-eb03-452a-bcf4-a5e91da83acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:06:44.621430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:44.618468    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:06:52.418479 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:52.415064    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w"]
Oct 30 17:06:52.418479 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:52.415427    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w" podUID="5d3e3542-3d7f-483d-9c05-49e437e9a930" containerName="maas-api" containerID="cri-o://ee2d8303888e2a390263ba26d1edf32539ff800009f16a5563539d439deee926" gracePeriod=30
Oct 30 17:06:53.014456 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:53.012231    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w"
Oct 30 17:06:53.077621 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:53.077580    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d3e3542-3d7f-483d-9c05-49e437e9a930-kube-api-access-dq5zx\" (UniqueName: \"kubernetes.io/projected/5d3e3542-3d7f-483d-9c05-49e437e9a930-kube-api-access-dq5zx\") pod \"5d3e3542-3d7f-483d-9c05-49e437e9a930\" (UID: \"5d3e3542-3d7f-483d-9c05-49e437e9a930\") "
Oct 30 17:06:53.078482 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:53.078456    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d3e3542-3d7f-483d-9c05-49e437e9a930-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d3e3542-3d7f-483d-9c05-49e437e9a930-metrics-certs\") pod \"5d3e3542-3d7f-483d-9c05-49e437e9a930\" (UID: \"5d3e3542-3d7f-483d-9c05-49e437e9a930\") "
Oct 30 17:06:53.080486 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:53.080465    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d3e3542-3d7f-483d-9c05-49e437e9a930-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d3e3542-3d7f-483d-9c05-49e437e9a930-maas-api-tls\") pod \"5d3e3542-3d7f-483d-9c05-49e437e9a930\" (UID: \"5d3e3542-3d7f-483d-9c05-49e437e9a930\") "
Oct 30 17:06:53.081077 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:53.081057    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d3e3542-3d7f-483d-9c05-49e437e9a930-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3e3542-3d7f-483d-9c05-49e437e9a930-openshift-service-ca\") pod \"5d3e3542-3d7f-483d-9c05-49e437e9a930\" (UID: \"5d3e3542-3d7f-483d-9c05-49e437e9a930\") "
Oct 30 17:06:53.081823 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:53.081712    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d3e3542-3d7f-483d-9c05-49e437e9a930-openshift-service-ca" pod "5d3e3542-3d7f-483d-9c05-49e437e9a930" (UID: "5d3e3542-3d7f-483d-9c05-49e437e9a930"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:06:53.082876 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:53.082759    4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3e3542-3d7f-483d-9c05-49e437e9a930-openshift-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:06:53.094567 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:53.091973    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d3e3542-3d7f-483d-9c05-49e437e9a930-metrics-certs" pod "5d3e3542-3d7f-483d-9c05-49e437e9a930" (UID: "5d3e3542-3d7f-483d-9c05-49e437e9a930"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:06:53.095474 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:53.094829    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d3e3542-3d7f-483d-9c05-49e437e9a930-maas-api-tls" pod "5d3e3542-3d7f-483d-9c05-49e437e9a930" (UID: "5d3e3542-3d7f-483d-9c05-49e437e9a930"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:06:53.112725 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:53.112691    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d3e3542-3d7f-483d-9c05-49e437e9a930-kube-api-access-dq5zx" pod "5d3e3542-3d7f-483d-9c05-49e437e9a930" (UID: "5d3e3542-3d7f-483d-9c05-49e437e9a930"). InnerVolumeSpecName "kube-api-access-dq5zx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:06:53.186456 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:53.184731    4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d3e3542-3d7f-483d-9c05-49e437e9a930-metrics-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:06:53.186456 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:53.184767    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d3e3542-3d7f-483d-9c05-49e437e9a930-maas-api-tls\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:06:53.186456 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:53.184782    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq5zx\" (UniqueName: \"kubernetes.io/projected/5d3e3542-3d7f-483d-9c05-49e437e9a930-kube-api-access-dq5zx\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:06:53.434317 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:53.434280    4433 generic.go:356] "Generic (PLEG): container finished" podID="5d3e3542-3d7f-483d-9c05-49e437e9a930" containerID="ee2d8303888e2a390263ba26d1edf32539ff800009f16a5563539d439deee926" exitCode=0
Oct 30 17:06:53.434771 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:53.434350    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w"
Oct 30 17:06:53.434771 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:53.434370    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w" event={"ID":"5d3e3542-3d7f-483d-9c05-49e437e9a930","Type":"ContainerDied","Data":"ee2d8303888e2a390263ba26d1edf32539ff800009f16a5563539d439deee926"}
Oct 30 17:06:53.434771 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:53.434476    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w" event={"ID":"5d3e3542-3d7f-483d-9c05-49e437e9a930","Type":"ContainerDied","Data":"79479fe38c8ece653f49d08d8ff0f599093bd92325a878b75f6605b1dfc94e0d"}
Oct 30 17:06:53.434771 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:53.434511    4433 scope.go:122] "RemoveContainer" containerID="ee2d8303888e2a390263ba26d1edf32539ff800009f16a5563539d439deee926"
Oct 30 17:06:53.455218 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:53.455190    4433 scope.go:122] "RemoveContainer" containerID="ee2d8303888e2a390263ba26d1edf32539ff800009f16a5563539d439deee926"
Oct 30 17:06:53.455630 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:06:53.455602    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee2d8303888e2a390263ba26d1edf32539ff800009f16a5563539d439deee926\": container with ID starting with ee2d8303888e2a390263ba26d1edf32539ff800009f16a5563539d439deee926 not found: ID does not exist" containerID="ee2d8303888e2a390263ba26d1edf32539ff800009f16a5563539d439deee926"
Oct 30 17:06:53.455726 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:53.455638    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee2d8303888e2a390263ba26d1edf32539ff800009f16a5563539d439deee926"} err="failed to get container status \"ee2d8303888e2a390263ba26d1edf32539ff800009f16a5563539d439deee926\": rpc error: code = NotFound desc = could not find container \"ee2d8303888e2a390263ba26d1edf32539ff800009f16a5563539d439deee926\": container with ID starting with ee2d8303888e2a390263ba26d1edf32539ff800009f16a5563539d439deee926 not found: ID does not exist"
Oct 30 17:06:53.499013 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:53.491945    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w"]
Oct 30 17:06:53.503885 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:53.503848    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1cdec0e0-6c4fff7bcb-8944w"]
Oct 30 17:06:53.697505 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:06:53.697364    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d3e3542-3d7f-483d-9c05-49e437e9a930" path="/var/lib/kubelet/pods/5d3e3542-3d7f-483d-9c05-49e437e9a930/volumes"
Oct 30 17:06:56.981260 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:06:56.981222    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:07:07.031667 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:07.031620    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d584bb3a-rhhmd"]
Oct 30 17:07:07.042783 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:07.042752    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d584bb3a-rhhmd"]
Oct 30 17:07:07.704813 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:07.704769    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51ecb056-722a-4cfd-a1bd-0d1ef679d7c3" path="/var/lib/kubelet/pods/51ecb056-722a-4cfd-a1bd-0d1ef679d7c3/volumes"
Oct 30 17:07:14.910582 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:14.910543    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aba8b6b-s2lk9"]
Oct 30 17:07:14.911209 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:14.911194    4433 cpu_manager.go:414] "containerMap: removing container" podUID="5d3e3542-3d7f-483d-9c05-49e437e9a930" containerName="maas-api"
Oct 30 17:07:14.911257 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:14.911211    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d3e3542-3d7f-483d-9c05-49e437e9a930" containerName="maas-api"
Oct 30 17:07:14.911325 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:14.911317    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d3e3542-3d7f-483d-9c05-49e437e9a930" containerName="maas-api"
Oct 30 17:07:14.916491 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:14.916466    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aba8b6b-s2lk9"
Oct 30 17:07:14.931139 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:14.931109    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:07:14.936291 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:14.936265    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aba8b6b-s2lk9"]
Oct 30 17:07:14.941853 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:14.941821    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\""
Oct 30 17:07:15.003902 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:15.003867    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5phsj\" (UniqueName: \"kubernetes.io/projected/cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999-kube-api-access-5phsj\") pod \"maas-api-revoke-keys-e2e-derive-4aba8b6b-s2lk9\" (UID: \"cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aba8b6b-s2lk9"
Oct 30 17:07:15.004110 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:15.004082    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-4aba8b6b-s2lk9\" (UID: \"cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aba8b6b-s2lk9"
Oct 30 17:07:15.004173 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:15.004134    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-4aba8b6b-s2lk9\" (UID: \"cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aba8b6b-s2lk9"
Oct 30 17:07:15.105991 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:15.105955    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5phsj\" (UniqueName: \"kubernetes.io/projected/cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999-kube-api-access-5phsj\") pod \"maas-api-revoke-keys-e2e-derive-4aba8b6b-s2lk9\" (UID: \"cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aba8b6b-s2lk9"
Oct 30 17:07:15.106187 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:15.106077    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-4aba8b6b-s2lk9\" (UID: \"cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aba8b6b-s2lk9"
Oct 30 17:07:15.106187 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:15.106099    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-4aba8b6b-s2lk9\" (UID: \"cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aba8b6b-s2lk9"
Oct 30 17:07:15.106874 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:15.106845    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-4aba8b6b-s2lk9\" (UID: \"cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aba8b6b-s2lk9"
Oct 30 17:07:15.109633 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:15.109525    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-4aba8b6b-s2lk9\" (UID: \"cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aba8b6b-s2lk9"
Oct 30 17:07:15.130971 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:15.130939    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5phsj\" (UniqueName: \"kubernetes.io/projected/cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999-kube-api-access-5phsj\") pod \"maas-api-revoke-keys-e2e-derive-4aba8b6b-s2lk9\" (UID: \"cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aba8b6b-s2lk9"
Oct 30 17:07:15.232817 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:15.232783    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aba8b6b-s2lk9"
Oct 30 17:07:15.547022 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:15.546998    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aba8b6b-s2lk9"]
Oct 30 17:07:15.549609 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:07:15.549575    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc4a5c0c_3f6d_4582_b8a9_bd8f9cf72999.slice/crio-87782732c6ab0c848dda20fc2e2a0676a015a993f9d9d268c6ca9c9a01bc14c2 WatchSource:0}: Error finding container 87782732c6ab0c848dda20fc2e2a0676a015a993f9d9d268c6ca9c9a01bc14c2: Status 404 returned error can't find the container with id 87782732c6ab0c848dda20fc2e2a0676a015a993f9d9d268c6ca9c9a01bc14c2
Oct 30 17:07:15.594469 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:15.594438    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aba8b6b-s2lk9" event={"ID":"cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999","Type":"ContainerStarted","Data":"87782732c6ab0c848dda20fc2e2a0676a015a993f9d9d268c6ca9c9a01bc14c2"}
Oct 30 17:07:16.732852 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:16.731702    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aba8b6b-s2lk9" event={"ID":"cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999","Type":"ContainerStarted","Data":"922d88a82ca9465c0c8088ffefe65efeef4bd4f47b648b551985ae426fd0c640"}
Oct 30 17:07:16.779664 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:16.779592    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aba8b6b-s2lk9" podStartSLOduration=2.779568513 podStartE2EDuration="2.779568513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:16.77260717 +0000 UTC m=+1213.207509403" watchObservedRunningTime="2026-10-30 17:07:16.779568513 +0000 UTC m=+1213.214470720"
Oct 30 17:07:17.845617 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:17.845565    4433 generic.go:356] "Generic (PLEG): container finished" podID="cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999" containerID="922d88a82ca9465c0c8088ffefe65efeef4bd4f47b648b551985ae426fd0c640" exitCode=0
Oct 30 17:07:17.846543 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:17.846515    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aba8b6b-s2lk9" event={"ID":"cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999","Type":"ContainerDied","Data":"922d88a82ca9465c0c8088ffefe65efeef4bd4f47b648b551985ae426fd0c640"}
Oct 30 17:07:18.563553 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:18.563499    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" podUID="458eba2e-eb03-452a-bcf4-a5e91da83acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:07:18.712865 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:18.712808    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 17:07:18.742564 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:18.742508    4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:07:18.770608 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:18.770563    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 30 17:07:18.932737 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:18.925437    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 17:07:18.941913 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:18.941866    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 30 17:07:19.006432 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:19.005503    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 17:07:19.197429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:19.184498    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:07:19.367453 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:19.364779    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm" podUID="bf43bf3d-f126-40a1-a24f-10214a85ceb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:07:19.583360 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:19.581508    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" podUID="458eba2e-eb03-452a-bcf4-a5e91da83acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:07:19.583360 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:19.582001    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:07:19.817447 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:19.698962    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 17:07:19.817447 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:19.727058    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:07:19.845776 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:19.836695    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:07:20.007429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:19.983504    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 17:07:20.007429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:19.984564    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:07:20.158939 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:20.136636    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:07:20.186430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:20.182418    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 17:07:20.206368 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:20.206334    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 17:07:20.206746 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:20.206725    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 17:07:20.244431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:20.227355    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"]
Oct 30 17:07:20.244431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:20.227491    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 17:07:20.244431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:20.229273    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 17:07:20.244431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:20.235683    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 17:07:20.284833 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:20.282896    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t" podUID="1f64def0-c36b-4a21-be70-77604220eaf1" containerName="istio-proxy" containerID="cri-o://7066935a5f5a1b936b007afbfc183f8f2376ccb73b7ea7772d07c7215ed853f7" gracePeriod=30
Oct 30 17:07:20.348203 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:20.348164    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aba8b6b-s2lk9"
Oct 30 17:07:20.375302 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:20.367829    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm" podUID="bf43bf3d-f126-40a1-a24f-10214a85ceb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:07:20.375302 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:20.369485    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 17:07:20.447320 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:20.447280    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999-kube-api-access-5phsj\" (UniqueName: \"kubernetes.io/projected/cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999-kube-api-access-5phsj\") pod \"cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999\" (UID: \"cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999\") "
Oct 30 17:07:20.447619 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:20.447593    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999-maas-api-xks-ca\") pod \"cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999\" (UID: \"cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999\") "
Oct 30 17:07:20.447711 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:20.447660    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999-maas-api-openshift-ca\") pod \"cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999\" (UID: \"cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999\") "
Oct 30 17:07:20.475443 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:20.468241    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999-maas-api-openshift-ca" pod "cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999" (UID: "cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:07:20.500495 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:20.495841    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999-maas-api-openshift-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:07:20.530669 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:20.530626    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999-kube-api-access-5phsj" pod "cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999" (UID: "cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999"). InnerVolumeSpecName "kube-api-access-5phsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:07:20.600751 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:20.599523    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" podUID="458eba2e-eb03-452a-bcf4-a5e91da83acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:07:20.662496 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:20.648735    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5phsj\" (UniqueName: \"kubernetes.io/projected/cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999-kube-api-access-5phsj\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:07:20.730429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:20.675094    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999-maas-api-xks-ca" pod "cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999" (UID: "cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:07:20.761879 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:20.761845    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999-maas-api-xks-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:07:20.763322 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:20.762582    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:07:20.815336 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:20.815265    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 17:07:20.976685 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:20.976650    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:07:20.991083 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:20.990903    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:07:20.991604 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:20.990903    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:07:21.013429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:20.992715    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 17:07:21.046631 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:21.046602    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 17:07:21.099137 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:21.099100    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 17:07:21.109776 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:21.109737    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:07:21.326540 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:21.326442    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aba8b6b-s2lk9" event={"ID":"cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999","Type":"ContainerDied","Data":"87782732c6ab0c848dda20fc2e2a0676a015a993f9d9d268c6ca9c9a01bc14c2"}
Oct 30 17:07:21.326824 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:21.326800    4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87782732c6ab0c848dda20fc2e2a0676a015a993f9d9d268c6ca9c9a01bc14c2"
Oct 30 17:07:21.330333 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:21.328043    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aba8b6b-s2lk9"
Oct 30 17:07:26.810543 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:26.810472    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"
Oct 30 17:07:26.901804 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:26.901769    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f64def0-c36b-4a21-be70-77604220eaf1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f64def0-c36b-4a21-be70-77604220eaf1-istio-podinfo\") pod \"1f64def0-c36b-4a21-be70-77604220eaf1\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") "
Oct 30 17:07:26.901987 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:26.901887    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f64def0-c36b-4a21-be70-77604220eaf1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f64def0-c36b-4a21-be70-77604220eaf1-credential-socket\") pod \"1f64def0-c36b-4a21-be70-77604220eaf1\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") "
Oct 30 17:07:26.902044 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:26.902024    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f64def0-c36b-4a21-be70-77604220eaf1-istio-token\" (UniqueName: \"kubernetes.io/projected/1f64def0-c36b-4a21-be70-77604220eaf1-istio-token\") pod \"1f64def0-c36b-4a21-be70-77604220eaf1\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") "
Oct 30 17:07:26.902112 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:26.902047    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f64def0-c36b-4a21-be70-77604220eaf1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f64def0-c36b-4a21-be70-77604220eaf1-workload-certs\") pod \"1f64def0-c36b-4a21-be70-77604220eaf1\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") "
Oct 30 17:07:26.902112 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:26.902073    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f64def0-c36b-4a21-be70-77604220eaf1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f64def0-c36b-4a21-be70-77604220eaf1-workload-socket\") pod \"1f64def0-c36b-4a21-be70-77604220eaf1\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") "
Oct 30 17:07:26.902223 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:26.902122    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f64def0-c36b-4a21-be70-77604220eaf1-kube-api-access-q58h5\" (UniqueName: \"kubernetes.io/projected/1f64def0-c36b-4a21-be70-77604220eaf1-kube-api-access-q58h5\") pod \"1f64def0-c36b-4a21-be70-77604220eaf1\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") "
Oct 30 17:07:26.902223 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:26.902144    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f64def0-c36b-4a21-be70-77604220eaf1-credential-socket" pod "1f64def0-c36b-4a21-be70-77604220eaf1" (UID: "1f64def0-c36b-4a21-be70-77604220eaf1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:07:26.902324 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:26.902236    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f64def0-c36b-4a21-be70-77604220eaf1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f64def0-c36b-4a21-be70-77604220eaf1-istiod-ca-cert\") pod \"1f64def0-c36b-4a21-be70-77604220eaf1\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") "
Oct 30 17:07:26.902381 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:26.902324    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f64def0-c36b-4a21-be70-77604220eaf1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f64def0-c36b-4a21-be70-77604220eaf1-istio-envoy\") pod \"1f64def0-c36b-4a21-be70-77604220eaf1\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") "
Oct 30 17:07:26.902480 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:26.902445    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f64def0-c36b-4a21-be70-77604220eaf1-workload-socket" pod "1f64def0-c36b-4a21-be70-77604220eaf1" (UID: "1f64def0-c36b-4a21-be70-77604220eaf1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:07:26.902544 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:26.902477    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f64def0-c36b-4a21-be70-77604220eaf1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f64def0-c36b-4a21-be70-77604220eaf1-istio-data\") pod \"1f64def0-c36b-4a21-be70-77604220eaf1\" (UID: \"1f64def0-c36b-4a21-be70-77604220eaf1\") "
Oct 30 17:07:26.902620 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:26.902592    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f64def0-c36b-4a21-be70-77604220eaf1-istiod-ca-cert" pod "1f64def0-c36b-4a21-be70-77604220eaf1" (UID: "1f64def0-c36b-4a21-be70-77604220eaf1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:07:26.902944 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:26.902923    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f64def0-c36b-4a21-be70-77604220eaf1-workload-certs" pod "1f64def0-c36b-4a21-be70-77604220eaf1" (UID: "1f64def0-c36b-4a21-be70-77604220eaf1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:07:26.903077 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:26.902959    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f64def0-c36b-4a21-be70-77604220eaf1-istio-data" pod "1f64def0-c36b-4a21-be70-77604220eaf1" (UID: "1f64def0-c36b-4a21-be70-77604220eaf1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:07:26.903198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:26.903163    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f64def0-c36b-4a21-be70-77604220eaf1-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:07:26.903198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:26.903181    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f64def0-c36b-4a21-be70-77604220eaf1-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:07:26.903198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:26.903195    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f64def0-c36b-4a21-be70-77604220eaf1-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:07:26.903351 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:26.903207    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f64def0-c36b-4a21-be70-77604220eaf1-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:07:26.903351 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:26.903215    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f64def0-c36b-4a21-be70-77604220eaf1-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:07:26.905030 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:26.904997    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f64def0-c36b-4a21-be70-77604220eaf1-istio-podinfo" pod "1f64def0-c36b-4a21-be70-77604220eaf1" (UID: "1f64def0-c36b-4a21-be70-77604220eaf1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:07:26.905030 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:26.905023    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f64def0-c36b-4a21-be70-77604220eaf1-kube-api-access-q58h5" pod "1f64def0-c36b-4a21-be70-77604220eaf1" (UID: "1f64def0-c36b-4a21-be70-77604220eaf1"). InnerVolumeSpecName "kube-api-access-q58h5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:07:26.905204 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:26.905002    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f64def0-c36b-4a21-be70-77604220eaf1-istio-envoy" pod "1f64def0-c36b-4a21-be70-77604220eaf1" (UID: "1f64def0-c36b-4a21-be70-77604220eaf1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:07:26.905204 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:26.905092    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f64def0-c36b-4a21-be70-77604220eaf1-istio-token" pod "1f64def0-c36b-4a21-be70-77604220eaf1" (UID: "1f64def0-c36b-4a21-be70-77604220eaf1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:07:27.009429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:27.004850    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f64def0-c36b-4a21-be70-77604220eaf1-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:07:27.009429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:27.004887    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q58h5\" (UniqueName: \"kubernetes.io/projected/1f64def0-c36b-4a21-be70-77604220eaf1-kube-api-access-q58h5\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:07:27.009429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:27.004903    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f64def0-c36b-4a21-be70-77604220eaf1-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:07:27.009429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:27.004915    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f64def0-c36b-4a21-be70-77604220eaf1-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:07:27.020586 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:07:27.010083    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:07:27.391361 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:27.391321    4433 generic.go:356] "Generic (PLEG): container finished" podID="1f64def0-c36b-4a21-be70-77604220eaf1" containerID="7066935a5f5a1b936b007afbfc183f8f2376ccb73b7ea7772d07c7215ed853f7" exitCode=0
Oct 30 17:07:27.391537 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:27.391371    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"
Oct 30 17:07:27.391537 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:27.391413    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t" event={"ID":"1f64def0-c36b-4a21-be70-77604220eaf1","Type":"ContainerDied","Data":"7066935a5f5a1b936b007afbfc183f8f2376ccb73b7ea7772d07c7215ed853f7"}
Oct 30 17:07:27.391537 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:27.391482    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t" event={"ID":"1f64def0-c36b-4a21-be70-77604220eaf1","Type":"ContainerDied","Data":"874ef052aa766628f55851479e8542eea54c3110c2b56fe3506cc6d13965b47e"}
Oct 30 17:07:27.391537 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:27.391512    4433 scope.go:122] "RemoveContainer" containerID="7066935a5f5a1b936b007afbfc183f8f2376ccb73b7ea7772d07c7215ed853f7"
Oct 30 17:07:27.402653 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:27.402630    4433 scope.go:122] "RemoveContainer" containerID="7066935a5f5a1b936b007afbfc183f8f2376ccb73b7ea7772d07c7215ed853f7"
Oct 30 17:07:27.402956 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:07:27.402939    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7066935a5f5a1b936b007afbfc183f8f2376ccb73b7ea7772d07c7215ed853f7\": container with ID starting with 7066935a5f5a1b936b007afbfc183f8f2376ccb73b7ea7772d07c7215ed853f7 not found: ID does not exist" containerID="7066935a5f5a1b936b007afbfc183f8f2376ccb73b7ea7772d07c7215ed853f7"
Oct 30 17:07:27.403006 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:27.402967    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7066935a5f5a1b936b007afbfc183f8f2376ccb73b7ea7772d07c7215ed853f7"} err="failed to get container status \"7066935a5f5a1b936b007afbfc183f8f2376ccb73b7ea7772d07c7215ed853f7\": rpc error: code = NotFound desc = could not find container \"7066935a5f5a1b936b007afbfc183f8f2376ccb73b7ea7772d07c7215ed853f7\": container with ID starting with 7066935a5f5a1b936b007afbfc183f8f2376ccb73b7ea7772d07c7215ed853f7 not found: ID does not exist"
Oct 30 17:07:27.444431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:27.444384    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"]
Oct 30 17:07:27.455991 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:27.455958    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-57s2t"]
Oct 30 17:07:27.696752 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:27.696718    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f64def0-c36b-4a21-be70-77604220eaf1" path="/var/lib/kubelet/pods/1f64def0-c36b-4a21-be70-77604220eaf1/volumes"
Oct 30 17:07:28.076262 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:28.076178    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v"]
Oct 30 17:07:28.076836 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:28.076818    4433 cpu_manager.go:414] "containerMap: removing container" podUID="cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999" containerName="revoke-keys"
Oct 30 17:07:28.076903 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:28.076838    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999" containerName="revoke-keys"
Oct 30 17:07:28.076903 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:28.076866    4433 cpu_manager.go:414] "containerMap: removing container" podUID="1f64def0-c36b-4a21-be70-77604220eaf1" containerName="istio-proxy"
Oct 30 17:07:28.076903 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:28.076872    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f64def0-c36b-4a21-be70-77604220eaf1" containerName="istio-proxy"
Oct 30 17:07:28.077050 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:28.076967    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f64def0-c36b-4a21-be70-77604220eaf1" containerName="istio-proxy"
Oct 30 17:07:28.077050 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:28.076978    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999" containerName="revoke-keys"
Oct 30 17:07:28.081848 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:28.081827    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v"
Oct 30 17:07:28.101906 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:28.101873    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v"]
Oct 30 17:07:28.108115 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:28.108085    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-5d8947-serving-cert\""
Oct 30 17:07:28.122332 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:28.122298    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b3d63c8-3b08-4283-a20a-7cfae44ae212-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v\" (UID: \"1b3d63c8-3b08-4283-a20a-7cfae44ae212\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v"
Oct 30 17:07:28.122590 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:28.122570    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b3d63c8-3b08-4283-a20a-7cfae44ae212-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v\" (UID: \"1b3d63c8-3b08-4283-a20a-7cfae44ae212\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v"
Oct 30 17:07:28.122719 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:28.122699    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b3d63c8-3b08-4283-a20a-7cfae44ae212-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v\" (UID: \"1b3d63c8-3b08-4283-a20a-7cfae44ae212\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v"
Oct 30 17:07:28.122810 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:28.122755    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvwdr\" (UniqueName: \"kubernetes.io/projected/1b3d63c8-3b08-4283-a20a-7cfae44ae212-kube-api-access-zvwdr\") pod \"maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v\" (UID: \"1b3d63c8-3b08-4283-a20a-7cfae44ae212\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v"
Oct 30 17:07:28.224255 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:28.224216    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b3d63c8-3b08-4283-a20a-7cfae44ae212-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v\" (UID: \"1b3d63c8-3b08-4283-a20a-7cfae44ae212\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v"
Oct 30 17:07:28.224519 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:28.224359    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b3d63c8-3b08-4283-a20a-7cfae44ae212-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v\" (UID: \"1b3d63c8-3b08-4283-a20a-7cfae44ae212\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v"
Oct 30 17:07:28.224519 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:07:28.224386    4433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5d8947-serving-cert: secret "maas-api-e2e-shared-a-w1-5d8947-serving-cert" not found
Oct 30 17:07:28.224659 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:07:28.224526    4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b3d63c8-3b08-4283-a20a-7cfae44ae212-maas-api-tls podName:1b3d63c8-3b08-4283-a20a-7cfae44ae212 nodeName:}" failed. No retries permitted until 2026-10-30 17:07:28.724497949 +0000 UTC m=+1225.159400146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1b3d63c8-3b08-4283-a20a-7cfae44ae212-maas-api-tls") pod "maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v" (UID: "1b3d63c8-3b08-4283-a20a-7cfae44ae212") : secret "maas-api-e2e-shared-a-w1-5d8947-serving-cert" not found
Oct 30 17:07:28.224659 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:28.224569    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b3d63c8-3b08-4283-a20a-7cfae44ae212-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v\" (UID: \"1b3d63c8-3b08-4283-a20a-7cfae44ae212\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v"
Oct 30 17:07:28.224659 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:28.224624    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvwdr\" (UniqueName: \"kubernetes.io/projected/1b3d63c8-3b08-4283-a20a-7cfae44ae212-kube-api-access-zvwdr\") pod \"maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v\" (UID: \"1b3d63c8-3b08-4283-a20a-7cfae44ae212\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v"
Oct 30 17:07:28.225248 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:28.225228    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b3d63c8-3b08-4283-a20a-7cfae44ae212-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v\" (UID: \"1b3d63c8-3b08-4283-a20a-7cfae44ae212\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v"
Oct 30 17:07:28.226890 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:28.226867    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b3d63c8-3b08-4283-a20a-7cfae44ae212-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v\" (UID: \"1b3d63c8-3b08-4283-a20a-7cfae44ae212\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v"
Oct 30 17:07:28.251552 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:28.251514    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvwdr\" (UniqueName: \"kubernetes.io/projected/1b3d63c8-3b08-4283-a20a-7cfae44ae212-kube-api-access-zvwdr\") pod \"maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v\" (UID: \"1b3d63c8-3b08-4283-a20a-7cfae44ae212\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v"
Oct 30 17:07:28.733848 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:28.733791    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b3d63c8-3b08-4283-a20a-7cfae44ae212-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v\" (UID: \"1b3d63c8-3b08-4283-a20a-7cfae44ae212\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v"
Oct 30 17:07:28.736480 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:28.736456    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b3d63c8-3b08-4283-a20a-7cfae44ae212-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v\" (UID: \"1b3d63c8-3b08-4283-a20a-7cfae44ae212\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v"
Oct 30 17:07:28.999429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:28.998732    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v"
Oct 30 17:07:29.345862 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:29.345825    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v"]
Oct 30 17:07:29.346532 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:07:29.346508    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b3d63c8_3b08_4283_a20a_7cfae44ae212.slice/crio-1503558f99d67000aec0be606459ea46913f0a781a35711fd402ee96b0b793cb WatchSource:0}: Error finding container 1503558f99d67000aec0be606459ea46913f0a781a35711fd402ee96b0b793cb: Status 404 returned error can't find the container with id 1503558f99d67000aec0be606459ea46913f0a781a35711fd402ee96b0b793cb
Oct 30 17:07:29.408873 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:29.408836    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v" event={"ID":"1b3d63c8-3b08-4283-a20a-7cfae44ae212","Type":"ContainerStarted","Data":"1503558f99d67000aec0be606459ea46913f0a781a35711fd402ee96b0b793cb"}
Oct 30 17:07:30.418736 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:30.418696    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v" event={"ID":"1b3d63c8-3b08-4283-a20a-7cfae44ae212","Type":"ContainerStarted","Data":"6b414db53ec388e198636ffcab1b3817bf05050aa87bf17b2cddc7b014b2d972"}
Oct 30 17:07:30.419150 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:30.418767    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v"
Oct 30 17:07:30.463641 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:30.463571    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v" podStartSLOduration=2.150472911 podStartE2EDuration="2.46353669s" totalImagesPullingTime="313.063779ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:30.461363039 +0000 UTC m=+1226.896265236" watchObservedRunningTime="2026-10-30 17:07:30.46353669 +0000 UTC m=+1226.898438906"
Oct 30 17:07:35.204932 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:35.204894    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aba8b6b-s2lk9"]
Oct 30 17:07:35.215792 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:35.215746    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4aba8b6b-s2lk9"]
Oct 30 17:07:35.697466 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:35.697427    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999" path="/var/lib/kubelet/pods/cc4a5c0c-3f6d-4582-b8a9-bd8f9cf72999/volumes"
Oct 30 17:07:36.433272 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:36.433243    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v"
Oct 30 17:07:48.786617 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:48.786561    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:07:50.011184 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.011130    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"]
Oct 30 17:07:50.059919 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.059886    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"]
Oct 30 17:07:50.060099 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.060021    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"
Oct 30 17:07:50.064727 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.064698    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"]
Oct 30 17:07:50.064863 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.064750    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"]
Oct 30 17:07:50.064863 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.064815    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"
Oct 30 17:07:50.172473 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.172367    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-wmkx2\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"
Oct 30 17:07:50.172473 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.172457    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/781b1fab-ca9f-4df7-8061-065561b69193-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zlfpc\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"
Oct 30 17:07:50.172720 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.172533    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/781b1fab-ca9f-4df7-8061-065561b69193-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zlfpc\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"
Oct 30 17:07:50.172720 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.172558    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-wmkx2\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"
Oct 30 17:07:50.172720 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.172581    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/781b1fab-ca9f-4df7-8061-065561b69193-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zlfpc\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"
Oct 30 17:07:50.172720 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.172605    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j7xp\" (UniqueName: \"kubernetes.io/projected/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-kube-api-access-2j7xp\") pod \"maas-default-gateway-openshift-default-6db586dd6c-wmkx2\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"
Oct 30 17:07:50.172720 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.172653    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-wmkx2\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"
Oct 30 17:07:50.172720 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.172686    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-wmkx2\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"
Oct 30 17:07:50.172720 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.172702    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcmth\" (UniqueName: \"kubernetes.io/projected/781b1fab-ca9f-4df7-8061-065561b69193-kube-api-access-dcmth\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zlfpc\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"
Oct 30 17:07:50.172720 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.172717    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/781b1fab-ca9f-4df7-8061-065561b69193-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zlfpc\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"
Oct 30 17:07:50.173159 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.172744    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/781b1fab-ca9f-4df7-8061-065561b69193-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zlfpc\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"
Oct 30 17:07:50.173159 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.172786    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/781b1fab-ca9f-4df7-8061-065561b69193-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zlfpc\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"
Oct 30 17:07:50.173159 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.172812    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-wmkx2\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"
Oct 30 17:07:50.173159 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.172829    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-wmkx2\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"
Oct 30 17:07:50.173159 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.172847    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-wmkx2\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"
Oct 30 17:07:50.173159 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.172891    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/781b1fab-ca9f-4df7-8061-065561b69193-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zlfpc\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"
Oct 30 17:07:50.173159 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.172938    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/781b1fab-ca9f-4df7-8061-065561b69193-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zlfpc\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"
Oct 30 17:07:50.173159 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.172958    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-wmkx2\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"
Oct 30 17:07:50.274068 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.273974    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/781b1fab-ca9f-4df7-8061-065561b69193-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zlfpc\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"
Oct 30 17:07:50.274068 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.274027    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-wmkx2\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"
Oct 30 17:07:50.274068 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.274056    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/781b1fab-ca9f-4df7-8061-065561b69193-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zlfpc\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"
Oct 30 17:07:50.274359 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.274092    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j7xp\" (UniqueName: \"kubernetes.io/projected/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-kube-api-access-2j7xp\") pod \"maas-default-gateway-openshift-default-6db586dd6c-wmkx2\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"
Oct 30 17:07:50.274359 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.274126    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-wmkx2\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"
Oct 30 17:07:50.274359 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.274164    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-wmkx2\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"
Oct 30 17:07:50.274359 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.274189    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcmth\" (UniqueName: \"kubernetes.io/projected/781b1fab-ca9f-4df7-8061-065561b69193-kube-api-access-dcmth\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zlfpc\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"
Oct 30 17:07:50.274359 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.274212    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/781b1fab-ca9f-4df7-8061-065561b69193-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zlfpc\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"
Oct 30 17:07:50.274359 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.274242    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/781b1fab-ca9f-4df7-8061-065561b69193-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zlfpc\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"
Oct 30 17:07:50.274359 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.274298    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/781b1fab-ca9f-4df7-8061-065561b69193-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zlfpc\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"
Oct 30 17:07:50.274359 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.274331    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-wmkx2\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"
Oct 30 17:07:50.274359 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.274354    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-wmkx2\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"
Oct 30 17:07:50.274831 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.274383    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-wmkx2\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"
Oct 30 17:07:50.274831 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.274461    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/781b1fab-ca9f-4df7-8061-065561b69193-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zlfpc\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"
Oct 30 17:07:50.274831 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.274523    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/781b1fab-ca9f-4df7-8061-065561b69193-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zlfpc\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"
Oct 30 17:07:50.274831 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.274555    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-wmkx2\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"
Oct 30 17:07:50.274831 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.274704    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-wmkx2\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"
Oct 30 17:07:50.274831 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.274734    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/781b1fab-ca9f-4df7-8061-065561b69193-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zlfpc\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"
Oct 30 17:07:50.276047 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.276001    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-wmkx2\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"
Oct 30 17:07:50.276546 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.276465    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-wmkx2\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"
Oct 30 17:07:50.277029 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.276772    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-wmkx2\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"
Oct 30 17:07:50.277029 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.276909    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/781b1fab-ca9f-4df7-8061-065561b69193-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zlfpc\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"
Oct 30 17:07:50.277029 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.276993    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-wmkx2\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"
Oct 30 17:07:50.277766 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.277737    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-wmkx2\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"
Oct 30 17:07:50.278071 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.278052    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/781b1fab-ca9f-4df7-8061-065561b69193-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zlfpc\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"
Oct 30 17:07:50.278375 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.278357    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/781b1fab-ca9f-4df7-8061-065561b69193-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zlfpc\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"
Oct 30 17:07:50.278737 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.278717    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/781b1fab-ca9f-4df7-8061-065561b69193-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zlfpc\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"
Oct 30 17:07:50.279189 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.279168    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/781b1fab-ca9f-4df7-8061-065561b69193-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zlfpc\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"
Oct 30 17:07:50.282052 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.281071    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-wmkx2\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"
Oct 30 17:07:50.282052 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.281437    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/781b1fab-ca9f-4df7-8061-065561b69193-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zlfpc\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"
Oct 30 17:07:50.282793 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.282764    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-wmkx2\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"
Oct 30 17:07:50.287790 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.287731    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/781b1fab-ca9f-4df7-8061-065561b69193-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zlfpc\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"
Oct 30 17:07:50.329478 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.319013    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j7xp\" (UniqueName: \"kubernetes.io/projected/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-kube-api-access-2j7xp\") pod \"maas-default-gateway-openshift-default-6db586dd6c-wmkx2\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"
Oct 30 17:07:50.329478 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.321475    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/781b1fab-ca9f-4df7-8061-065561b69193-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zlfpc\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"
Oct 30 17:07:50.329478 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.328078    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-wmkx2\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"
Oct 30 17:07:50.329478 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.329111    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcmth\" (UniqueName: \"kubernetes.io/projected/781b1fab-ca9f-4df7-8061-065561b69193-kube-api-access-dcmth\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zlfpc\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"
Oct 30 17:07:50.381799 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.379724    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"
Oct 30 17:07:50.391827 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.391727    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"
Oct 30 17:07:50.657388 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.657304    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d4f99-8ff54d7d7-qp8mh"]
Oct 30 17:07:50.662963 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.662942    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d4f99-8ff54d7d7-qp8mh"
Oct 30 17:07:50.678909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.678859    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-6d4f99\""
Oct 30 17:07:50.682416 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.682364    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2lcq\" (UniqueName: \"kubernetes.io/projected/7741e951-b0c5-432a-ae92-8dc492b47cf7-kube-api-access-b2lcq\") pod \"payload-pre-processing-e2e-shared-b-w2-6d4f99-8ff54d7d7-qp8mh\" (UID: \"7741e951-b0c5-432a-ae92-8dc492b47cf7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d4f99-8ff54d7d7-qp8mh"
Oct 30 17:07:50.682739 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.682703    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7741e951-b0c5-432a-ae92-8dc492b47cf7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-6d4f99-8ff54d7d7-qp8mh\" (UID: \"7741e951-b0c5-432a-ae92-8dc492b47cf7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d4f99-8ff54d7d7-qp8mh"
Oct 30 17:07:50.685015 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.684973    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d4f99-8ff54d7d7-qp8mh"]
Oct 30 17:07:50.720339 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.719982    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-6d4f99-dockercfg-jtbbp\""
Oct 30 17:07:50.738766 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.738719    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"]
Oct 30 17:07:50.741235 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:07:50.741203    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf3ba2d5_bc3c_48dc_a8fe_b3d850ea2eb3.slice/crio-8977016f57cccfc42e452ef4fdb74b1b9ff68992c277b9c60fa2852ff43bd9c5 WatchSource:0}: Error finding container 8977016f57cccfc42e452ef4fdb74b1b9ff68992c277b9c60fa2852ff43bd9c5: Status 404 returned error can't find the container with id 8977016f57cccfc42e452ef4fdb74b1b9ff68992c277b9c60fa2852ff43bd9c5
Oct 30 17:07:50.744285 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.744246    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:07:50.744441 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.744341    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:07:50.744441 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.744383    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:07:50.759390 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.759253    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"]
Oct 30 17:07:50.762094 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:07:50.762034    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod781b1fab_ca9f_4df7_8061_065561b69193.slice/crio-6f34e11b59e01c2a9ba057a239d8139ee86d86d93bca31e7f5b120618ce0cae5 WatchSource:0}: Error finding container 6f34e11b59e01c2a9ba057a239d8139ee86d86d93bca31e7f5b120618ce0cae5: Status 404 returned error can't find the container with id 6f34e11b59e01c2a9ba057a239d8139ee86d86d93bca31e7f5b120618ce0cae5
Oct 30 17:07:50.765214 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.765167    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:07:50.765332 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.765263    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:07:50.765332 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.765316    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:07:50.784344 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.784314    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7741e951-b0c5-432a-ae92-8dc492b47cf7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-6d4f99-8ff54d7d7-qp8mh\" (UID: \"7741e951-b0c5-432a-ae92-8dc492b47cf7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d4f99-8ff54d7d7-qp8mh"
Oct 30 17:07:50.784713 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.784649    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2lcq\" (UniqueName: \"kubernetes.io/projected/7741e951-b0c5-432a-ae92-8dc492b47cf7-kube-api-access-b2lcq\") pod \"payload-pre-processing-e2e-shared-b-w2-6d4f99-8ff54d7d7-qp8mh\" (UID: \"7741e951-b0c5-432a-ae92-8dc492b47cf7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d4f99-8ff54d7d7-qp8mh"
Oct 30 17:07:50.785290 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.785255    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7741e951-b0c5-432a-ae92-8dc492b47cf7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-6d4f99-8ff54d7d7-qp8mh\" (UID: \"7741e951-b0c5-432a-ae92-8dc492b47cf7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d4f99-8ff54d7d7-qp8mh"
Oct 30 17:07:50.810109 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.810075    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2lcq\" (UniqueName: \"kubernetes.io/projected/7741e951-b0c5-432a-ae92-8dc492b47cf7-kube-api-access-b2lcq\") pod \"payload-pre-processing-e2e-shared-b-w2-6d4f99-8ff54d7d7-qp8mh\" (UID: \"7741e951-b0c5-432a-ae92-8dc492b47cf7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d4f99-8ff54d7d7-qp8mh"
Oct 30 17:07:50.979428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:50.979202    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d4f99-8ff54d7d7-qp8mh"
Oct 30 17:07:51.324363 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:51.324259    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf"]
Oct 30 17:07:51.342724 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:51.342693    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf"]
Oct 30 17:07:51.343045 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:51.342986    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf"
Oct 30 17:07:51.363619 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:51.363588    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7ebeb6e1-serving-cert\""
Oct 30 17:07:51.403508 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:51.403104    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d4f99-8ff54d7d7-qp8mh"]
Oct 30 17:07:51.407000 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:07:51.406815    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7741e951_b0c5_432a_ae92_8dc492b47cf7.slice/crio-8dbe06d8c141ca60da97a3dd4889b49294af69c91f1756abc1eac3af0cb78b2f WatchSource:0}: Error finding container 8dbe06d8c141ca60da97a3dd4889b49294af69c91f1756abc1eac3af0cb78b2f: Status 404 returned error can't find the container with id 8dbe06d8c141ca60da97a3dd4889b49294af69c91f1756abc1eac3af0cb78b2f
Oct 30 17:07:51.529749 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:51.528847    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7761df4-6220-4f57-ac52-ffe62202337d-metrics-certs\") pod \"maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf\" (UID: \"e7761df4-6220-4f57-ac52-ffe62202337d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf"
Oct 30 17:07:51.529749 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:51.528895    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr5vq\" (UniqueName: \"kubernetes.io/projected/e7761df4-6220-4f57-ac52-ffe62202337d-kube-api-access-sr5vq\") pod \"maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf\" (UID: \"e7761df4-6220-4f57-ac52-ffe62202337d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf"
Oct 30 17:07:51.529749 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:51.528987    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7761df4-6220-4f57-ac52-ffe62202337d-maas-api-tls\") pod \"maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf\" (UID: \"e7761df4-6220-4f57-ac52-ffe62202337d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf"
Oct 30 17:07:51.529749 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:51.529246    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7761df4-6220-4f57-ac52-ffe62202337d-openshift-service-ca\") pod \"maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf\" (UID: \"e7761df4-6220-4f57-ac52-ffe62202337d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf"
Oct 30 17:07:51.598673 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:51.598577    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2" event={"ID":"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3","Type":"ContainerStarted","Data":"d01ac2962194a9324768421c0b8f400f64402951bf5594094e29126112894300"}
Oct 30 17:07:51.598926 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:51.598906    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2" event={"ID":"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3","Type":"ContainerStarted","Data":"8977016f57cccfc42e452ef4fdb74b1b9ff68992c277b9c60fa2852ff43bd9c5"}
Oct 30 17:07:51.601848 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:51.601817    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d4f99-8ff54d7d7-qp8mh" event={"ID":"7741e951-b0c5-432a-ae92-8dc492b47cf7","Type":"ContainerStarted","Data":"8dbe06d8c141ca60da97a3dd4889b49294af69c91f1756abc1eac3af0cb78b2f"}
Oct 30 17:07:51.604148 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:51.604125    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc" event={"ID":"781b1fab-ca9f-4df7-8061-065561b69193","Type":"ContainerStarted","Data":"3b7eec95adcfffdad27c9468aefa24a61224b90e3792eb2b4a63fa77039faabe"}
Oct 30 17:07:51.605153 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:51.605125    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc" event={"ID":"781b1fab-ca9f-4df7-8061-065561b69193","Type":"ContainerStarted","Data":"6f34e11b59e01c2a9ba057a239d8139ee86d86d93bca31e7f5b120618ce0cae5"}
Oct 30 17:07:51.638432 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:51.638372    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7761df4-6220-4f57-ac52-ffe62202337d-openshift-service-ca\") pod \"maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf\" (UID: \"e7761df4-6220-4f57-ac52-ffe62202337d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf"
Oct 30 17:07:51.638695 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:51.638682    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7761df4-6220-4f57-ac52-ffe62202337d-metrics-certs\") pod \"maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf\" (UID: \"e7761df4-6220-4f57-ac52-ffe62202337d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf"
Oct 30 17:07:51.638812 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:51.638799    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr5vq\" (UniqueName: \"kubernetes.io/projected/e7761df4-6220-4f57-ac52-ffe62202337d-kube-api-access-sr5vq\") pod \"maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf\" (UID: \"e7761df4-6220-4f57-ac52-ffe62202337d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf"
Oct 30 17:07:51.639009 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:51.638999    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7761df4-6220-4f57-ac52-ffe62202337d-maas-api-tls\") pod \"maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf\" (UID: \"e7761df4-6220-4f57-ac52-ffe62202337d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf"
Oct 30 17:07:51.639545 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:51.639518    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7761df4-6220-4f57-ac52-ffe62202337d-openshift-service-ca\") pod \"maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf\" (UID: \"e7761df4-6220-4f57-ac52-ffe62202337d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf"
Oct 30 17:07:51.646038 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:51.645976    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7761df4-6220-4f57-ac52-ffe62202337d-metrics-certs\") pod \"maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf\" (UID: \"e7761df4-6220-4f57-ac52-ffe62202337d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf"
Oct 30 17:07:51.646654 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:51.646635    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7761df4-6220-4f57-ac52-ffe62202337d-maas-api-tls\") pod \"maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf\" (UID: \"e7761df4-6220-4f57-ac52-ffe62202337d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf"
Oct 30 17:07:51.667437 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:51.660820    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2" podStartSLOduration=2.660799162 podStartE2EDuration="2.660799162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:51.658457755 +0000 UTC m=+1248.093359961" watchObservedRunningTime="2026-10-30 17:07:51.660799162 +0000 UTC m=+1248.095701369"
Oct 30 17:07:51.770451 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:51.742958    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr5vq\" (UniqueName: \"kubernetes.io/projected/e7761df4-6220-4f57-ac52-ffe62202337d-kube-api-access-sr5vq\") pod \"maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf\" (UID: \"e7761df4-6220-4f57-ac52-ffe62202337d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf"
Oct 30 17:07:51.779069 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:51.779006    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc" podStartSLOduration=2.7789855599999997 podStartE2EDuration="2.77898556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:51.751910698 +0000 UTC m=+1248.186812904" watchObservedRunningTime="2026-10-30 17:07:51.77898556 +0000 UTC m=+1248.213887768"
Oct 30 17:07:52.024462 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:52.000259    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf"
Oct 30 17:07:52.777759 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:52.763822    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"
Oct 30 17:07:52.777759 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:52.763888    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"
Oct 30 17:07:52.852448 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:52.851943    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zlfpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 17:07:52.852448 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:52.852024    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc" podUID="781b1fab-ca9f-4df7-8061-065561b69193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 17:07:52.852448 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:52.852438    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-wmkx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 17:07:52.852801 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:52.852473    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2" podUID="df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 17:07:53.298437 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:53.282986    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf"]
Oct 30 17:07:53.393550 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:53.392796    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zlfpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 17:07:53.393550 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:53.392855    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc" podUID="781b1fab-ca9f-4df7-8061-065561b69193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 17:07:53.393550 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:53.393492    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-wmkx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 17:07:53.393550 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:53.393535    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2" podUID="df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 17:07:53.785431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:53.765493    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" podUID="458eba2e-eb03-452a-bcf4-a5e91da83acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:07:53.922574 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:53.918372    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:07:53.922574 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:53.919578    4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": context deadline exceeded"
Oct 30 17:07:53.922574 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:53.919650    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 17:07:54.003456 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:54.001080    4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:07:54.028480 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:54.013301    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm" podUID="bf43bf3d-f126-40a1-a24f-10214a85ceb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:07:54.121742 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:54.121609    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:07:54.143052 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:54.143004    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 30 17:07:54.265626 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:54.254140    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:07:54.394493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:54.392352    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-wmkx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 17:07:54.394493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:54.392920    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zlfpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 17:07:54.394493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:54.392982    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc" podUID="781b1fab-ca9f-4df7-8061-065561b69193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 17:07:54.400692 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:54.400558    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2" podUID="df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 17:07:54.487388 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:54.484271    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf" event={"ID":"e7761df4-6220-4f57-ac52-ffe62202337d","Type":"ContainerStarted","Data":"eb9e35ce4a2733126aad84c78758331166e9a330ba40a33437059c4d8f7a8c4e"}
Oct 30 17:07:54.502084 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:54.491445    4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm" podUID="bf43bf3d-f126-40a1-a24f-10214a85ceb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:07:54.765700 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:54.764511    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 17:07:54.928792 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:54.925547    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 17:07:54.928792 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:54.926323    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:07:55.076487 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:55.058389    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:07:55.076487 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:55.066494    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm" podUID="bf43bf3d-f126-40a1-a24f-10214a85ceb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:07:55.076487 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:55.066939    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 17:07:55.103683 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:55.102663    4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:07:55.150450 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:55.122564    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:07:55.150450 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:55.123048    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:07:55.277451 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:55.274638    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:07:55.329503 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:55.279105    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 17:07:55.474246 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:55.473723    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 17:07:55.502322 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:55.502266    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-wmkx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 17:07:55.502713 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:55.502669    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2" podUID="df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 17:07:55.511729 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:55.511699    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zlfpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 17:07:55.515563 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:55.515529    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc" podUID="781b1fab-ca9f-4df7-8061-065561b69193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 17:07:55.563539 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:55.563481    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf" event={"ID":"e7761df4-6220-4f57-ac52-ffe62202337d","Type":"ContainerStarted","Data":"94b06c7645a6d28cb347af6d6a3a4421c723593943d5b09e8546a31b99aa7c18"}
Oct 30 17:07:55.601542 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:55.601445    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf"
Oct 30 17:07:55.862482 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:55.830609    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 17:07:55.972371 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:55.869647    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:07:56.045793 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:55.940135    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:07:56.076839 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:56.076798    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 17:07:56.078923 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:56.078890    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm" podUID="bf43bf3d-f126-40a1-a24f-10214a85ceb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:07:56.117502 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:56.112800    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:07:56.117502 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:56.112858    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 17:07:56.117502 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:56.116254    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:07:56.123556 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:56.123511    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:07:56.349536 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:56.322729    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:07:56.375536 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:56.349829    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:07:56.399634 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:56.399596    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-wmkx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 17:07:56.399984 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:56.399952    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2" podUID="df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 17:07:56.406428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:56.402832    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zlfpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 17:07:56.468634 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:56.468590    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc" podUID="781b1fab-ca9f-4df7-8061-065561b69193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 17:07:56.478165 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:56.478103    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf" podStartSLOduration=4.738302166 podStartE2EDuration="5.478083616s" totalImagesPullingTime="739.78145ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:55.629693991 +0000 UTC m=+1252.064596214" watchObservedRunningTime="2026-10-30 17:07:56.478083616 +0000 UTC m=+1252.912985834"
Oct 30 17:07:56.525068 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:56.524080    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 17:07:56.629120 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:56.628964    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:07:56.688427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:56.649859    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 17:07:57.042044 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:07:57.041697    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:07:57.389154 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:57.388698    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-wmkx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 17:07:57.389154 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:57.388759    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2" podUID="df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 17:07:57.394563 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:57.394448    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zlfpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 17:07:57.394563 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:57.394502    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc" podUID="781b1fab-ca9f-4df7-8061-065561b69193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 17:07:58.384338 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:58.384294    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-wmkx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 17:07:58.384737 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:58.384367    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2" podUID="df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 17:07:58.393054 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:58.393015    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zlfpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 17:07:58.393190 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:58.393082    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc" podUID="781b1fab-ca9f-4df7-8061-065561b69193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 17:07:59.385039 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:59.385002    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-wmkx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 17:07:59.385556 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:59.385065    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2" podUID="df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 17:07:59.392503 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:59.392345    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zlfpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 17:07:59.392503 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:59.392460    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc" podUID="781b1fab-ca9f-4df7-8061-065561b69193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 17:07:59.763117 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:59.763077    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d4f99-8ff54d7d7-qp8mh" event={"ID":"7741e951-b0c5-432a-ae92-8dc492b47cf7","Type":"ContainerStarted","Data":"0cc3129a188de66ea5439687f2cb29905f6ac21823ed69b5a1a325291a4060e9"}
Oct 30 17:07:59.763316 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:59.763187    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d4f99-8ff54d7d7-qp8mh"
Oct 30 17:07:59.800464 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:59.799991    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"]
Oct 30 17:07:59.813547 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:59.813511    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:07:59.818344 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:59.817340    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d4f99-8ff54d7d7-qp8mh" podStartSLOduration=1.6302691280000001 podStartE2EDuration="9.817321646s" totalImagesPullingTime="8.187052518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:59.80751908 +0000 UTC m=+1256.242421289" watchObservedRunningTime="2026-10-30 17:07:59.817321646 +0000 UTC m=+1256.252223854"
Oct 30 17:07:59.830336 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:59.829186    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-a43843-openshift-default-dockercfg-gxd57\""
Oct 30 17:07:59.880564 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:59.880519    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"]
Oct 30 17:07:59.912296 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:59.912196    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4f0b0a7-7c36-4ad7-8573-582828ad701b-workload-socket\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:07:59.912505 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:59.912340    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4f0b0a7-7c36-4ad7-8573-582828ad701b-istio-podinfo\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:07:59.912586 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:59.912506    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4f0b0a7-7c36-4ad7-8573-582828ad701b-credential-socket\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:07:59.912632 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:59.912581    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4f0b0a7-7c36-4ad7-8573-582828ad701b-istio-data\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:07:59.912714 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:59.912687    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4f0b0a7-7c36-4ad7-8573-582828ad701b-workload-certs\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:07:59.912754 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:59.912744    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4f0b0a7-7c36-4ad7-8573-582828ad701b-istio-envoy\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:07:59.912971 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:59.912953    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4f0b0a7-7c36-4ad7-8573-582828ad701b-istiod-ca-cert\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:07:59.913047 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:59.913007    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lvql\" (UniqueName: \"kubernetes.io/projected/e4f0b0a7-7c36-4ad7-8573-582828ad701b-kube-api-access-2lvql\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:07:59.913139 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:07:59.913125    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4f0b0a7-7c36-4ad7-8573-582828ad701b-istio-token\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:08:00.028425 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:00.027511    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4f0b0a7-7c36-4ad7-8573-582828ad701b-workload-socket\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:08:00.028425 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:00.027607    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4f0b0a7-7c36-4ad7-8573-582828ad701b-istio-podinfo\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:08:00.028425 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:00.027663    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4f0b0a7-7c36-4ad7-8573-582828ad701b-credential-socket\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:08:00.028425 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:00.027733    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4f0b0a7-7c36-4ad7-8573-582828ad701b-istio-data\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:08:00.028425 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:00.027775    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4f0b0a7-7c36-4ad7-8573-582828ad701b-workload-certs\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:08:00.028425 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:00.027825    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4f0b0a7-7c36-4ad7-8573-582828ad701b-istio-envoy\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:08:00.028425 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:00.027965    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4f0b0a7-7c36-4ad7-8573-582828ad701b-istiod-ca-cert\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:08:00.028425 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:00.028021    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lvql\" (UniqueName: \"kubernetes.io/projected/e4f0b0a7-7c36-4ad7-8573-582828ad701b-kube-api-access-2lvql\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:08:00.028425 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:00.028115    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4f0b0a7-7c36-4ad7-8573-582828ad701b-istio-token\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:08:00.029056 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:00.028981    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4f0b0a7-7c36-4ad7-8573-582828ad701b-workload-socket\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:08:00.040438 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:00.029726    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4f0b0a7-7c36-4ad7-8573-582828ad701b-workload-certs\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:08:00.040438 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:00.029988    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4f0b0a7-7c36-4ad7-8573-582828ad701b-credential-socket\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:08:00.040438 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:00.030247    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4f0b0a7-7c36-4ad7-8573-582828ad701b-istio-data\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:08:00.040438 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:00.031178    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4f0b0a7-7c36-4ad7-8573-582828ad701b-istiod-ca-cert\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:08:00.040438 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:00.038458    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4f0b0a7-7c36-4ad7-8573-582828ad701b-istio-podinfo\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:08:00.069424 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:00.067757    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4f0b0a7-7c36-4ad7-8573-582828ad701b-istio-envoy\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:08:00.101427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:00.091278    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4f0b0a7-7c36-4ad7-8573-582828ad701b-istio-token\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:08:00.101427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:00.101203    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lvql\" (UniqueName: \"kubernetes.io/projected/e4f0b0a7-7c36-4ad7-8573-582828ad701b-kube-api-access-2lvql\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:08:00.142429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:00.140675    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:08:00.415431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:00.408739    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-wmkx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 17:08:00.415431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:00.408833    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2" podUID="df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 17:08:00.415431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:00.409259    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zlfpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 17:08:00.415431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:00.409300    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc" podUID="781b1fab-ca9f-4df7-8061-065561b69193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 17:08:00.615703 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:00.615454    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"]
Oct 30 17:08:00.624863 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:00.624811    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:08:00.624992 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:00.624899    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:08:00.624992 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:00.624942    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:08:00.776815 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:00.776780    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb" event={"ID":"e4f0b0a7-7c36-4ad7-8573-582828ad701b","Type":"ContainerStarted","Data":"2c1e5d2ddff5a0807279bf736c587e972339e07ddfeb32c620483e4d929ffe24"}
Oct 30 17:08:00.777001 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:00.776833    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb" event={"ID":"e4f0b0a7-7c36-4ad7-8573-582828ad701b","Type":"ContainerStarted","Data":"bb9dc68ef4fd15822bb1204d8e33cfc2dd8e815e8969d2ffee04b22a53eb8f8f"}
Oct 30 17:08:01.145761 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:01.145634    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:08:01.145761 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:01.145760    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 17:08:01.146013 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:01.145848    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb" podUID="e4f0b0a7-7c36-4ad7-8573-582828ad701b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 17:08:01.392614 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:01.392574    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zlfpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 17:08:01.392837 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:01.392654    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc" podUID="781b1fab-ca9f-4df7-8061-065561b69193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 17:08:01.743459 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:01.739520    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"
Oct 30 17:08:01.743459 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:01.740294    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"
Oct 30 17:08:01.743459 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:01.741953    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf"
Oct 30 17:08:01.743459 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:01.742384    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"
Oct 30 17:08:01.788115 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:01.786612    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb" podStartSLOduration=2.786588178 podStartE2EDuration="2.786588178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:08:00.832631234 +0000 UTC m=+1257.267533441" watchObservedRunningTime="2026-10-30 17:08:01.786588178 +0000 UTC m=+1258.221490385"
Oct 30 17:08:02.145951 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:02.145870    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:08:02.576336 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:02.576304    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"
Oct 30 17:08:02.576853 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:02.576827    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"
Oct 30 17:08:02.578197 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:02.578177    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"
Oct 30 17:08:02.807203 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:02.806766    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:08:02.811150 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:02.811119    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:08:04.144968 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:04.144937    4433 scope.go:122] "RemoveContainer" containerID="f1ccb5bd653a9f08e9850696b0583acb0f6387e183f7e128f204b69e53ada26b"
Oct 30 17:08:04.155289 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:04.155265    4433 scope.go:122] "RemoveContainer" containerID="6420f1bff3dbc7c2b261cb24dd8a07bfcf4adf7717747a73b288d7c0bd41fd34"
Oct 30 17:08:04.164948 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:04.164921    4433 scope.go:122] "RemoveContainer" containerID="b20b0434a6779f90a476be2d840035542b4d48f92c58e70c41d16a8811adf19d"
Oct 30 17:08:04.174308 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:04.174285    4433 scope.go:122] "RemoveContainer" containerID="166c8f0fe5463c968aabb39a7d0200d5420dc37792f11d7a2dedcaed2765fc44"
Oct 30 17:08:04.183694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:04.183672    4433 scope.go:122] "RemoveContainer" containerID="7b380428a0e3c1a487c07b65ca00e74224f6f410daba9dd31da7c650503ca23a"
Oct 30 17:08:04.193507 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:04.193483    4433 scope.go:122] "RemoveContainer" containerID="b41023255b0261bbc786418db05a71ff1fe0e04b0843663de3f5abceb958c6a2"
Oct 30 17:08:10.779603 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:10.779571    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d4f99-8ff54d7d7-qp8mh"
Oct 30 17:08:14.296021 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:14.295982    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw"]
Oct 30 17:08:14.302553 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:14.302533    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw"
Oct 30 17:08:14.330301 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:14.330262    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-6d4f99-serving-cert\""
Oct 30 17:08:14.336931 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:14.336901    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw"]
Oct 30 17:08:14.448253 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:14.448216    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85c6e2b8-220e-4493-9468-e15d18966ae5-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw\" (UID: \"85c6e2b8-220e-4493-9468-e15d18966ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw"
Oct 30 17:08:14.448491 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:14.448453    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85c6e2b8-220e-4493-9468-e15d18966ae5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw\" (UID: \"85c6e2b8-220e-4493-9468-e15d18966ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw"
Oct 30 17:08:14.448554 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:14.448520    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfd85\" (UniqueName: \"kubernetes.io/projected/85c6e2b8-220e-4493-9468-e15d18966ae5-kube-api-access-bfd85\") pod \"maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw\" (UID: \"85c6e2b8-220e-4493-9468-e15d18966ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw"
Oct 30 17:08:14.448651 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:14.448625    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85c6e2b8-220e-4493-9468-e15d18966ae5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw\" (UID: \"85c6e2b8-220e-4493-9468-e15d18966ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw"
Oct 30 17:08:14.550595 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:14.550519    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85c6e2b8-220e-4493-9468-e15d18966ae5-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw\" (UID: \"85c6e2b8-220e-4493-9468-e15d18966ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw"
Oct 30 17:08:14.550739 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:14.550652    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85c6e2b8-220e-4493-9468-e15d18966ae5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw\" (UID: \"85c6e2b8-220e-4493-9468-e15d18966ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw"
Oct 30 17:08:14.550739 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:14.550695    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfd85\" (UniqueName: \"kubernetes.io/projected/85c6e2b8-220e-4493-9468-e15d18966ae5-kube-api-access-bfd85\") pod \"maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw\" (UID: \"85c6e2b8-220e-4493-9468-e15d18966ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw"
Oct 30 17:08:14.550828 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:08:14.550806    4433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d4f99-serving-cert: secret "maas-api-e2e-shared-b-w2-6d4f99-serving-cert" not found
Oct 30 17:08:14.550871 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:14.550820    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85c6e2b8-220e-4493-9468-e15d18966ae5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw\" (UID: \"85c6e2b8-220e-4493-9468-e15d18966ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw"
Oct 30 17:08:14.550928 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:08:14.550922    4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85c6e2b8-220e-4493-9468-e15d18966ae5-maas-api-tls podName:85c6e2b8-220e-4493-9468-e15d18966ae5 nodeName:}" failed. No retries permitted until 2026-10-30 17:08:15.050900173 +0000 UTC m=+1271.485802358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/85c6e2b8-220e-4493-9468-e15d18966ae5-maas-api-tls") pod "maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw" (UID: "85c6e2b8-220e-4493-9468-e15d18966ae5") : secret "maas-api-e2e-shared-b-w2-6d4f99-serving-cert" not found
Oct 30 17:08:14.551486 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:14.551461    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85c6e2b8-220e-4493-9468-e15d18966ae5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw\" (UID: \"85c6e2b8-220e-4493-9468-e15d18966ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw"
Oct 30 17:08:14.553108 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:14.553092    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85c6e2b8-220e-4493-9468-e15d18966ae5-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw\" (UID: \"85c6e2b8-220e-4493-9468-e15d18966ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw"
Oct 30 17:08:14.574522 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:14.574488    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfd85\" (UniqueName: \"kubernetes.io/projected/85c6e2b8-220e-4493-9468-e15d18966ae5-kube-api-access-bfd85\") pod \"maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw\" (UID: \"85c6e2b8-220e-4493-9468-e15d18966ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw"
Oct 30 17:08:15.058542 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:15.058501    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85c6e2b8-220e-4493-9468-e15d18966ae5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw\" (UID: \"85c6e2b8-220e-4493-9468-e15d18966ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw"
Oct 30 17:08:15.061059 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:15.061033    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85c6e2b8-220e-4493-9468-e15d18966ae5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw\" (UID: \"85c6e2b8-220e-4493-9468-e15d18966ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw"
Oct 30 17:08:15.222433 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:15.221052    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw"
Oct 30 17:08:15.594056 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:15.594024    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw"]
Oct 30 17:08:15.595344 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:08:15.595312    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85c6e2b8_220e_4493_9468_e15d18966ae5.slice/crio-266bad1c14231d02f198cf79c0ea08dc26c3fd5ad5376b3e08e66aba51e1fa46 WatchSource:0}: Error finding container 266bad1c14231d02f198cf79c0ea08dc26c3fd5ad5376b3e08e66aba51e1fa46: Status 404 returned error can't find the container with id 266bad1c14231d02f198cf79c0ea08dc26c3fd5ad5376b3e08e66aba51e1fa46
Oct 30 17:08:15.900274 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:15.900191    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw" event={"ID":"85c6e2b8-220e-4493-9468-e15d18966ae5","Type":"ContainerStarted","Data":"266bad1c14231d02f198cf79c0ea08dc26c3fd5ad5376b3e08e66aba51e1fa46"}
Oct 30 17:08:16.908736 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:16.908698    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw" event={"ID":"85c6e2b8-220e-4493-9468-e15d18966ae5","Type":"ContainerStarted","Data":"d06ada848dfcb340357207d468747a840c1f111aabdb6fe557d62edf3e09adb3"}
Oct 30 17:08:16.909159 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:16.908840    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw"
Oct 30 17:08:16.954136 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:16.954080    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw" podStartSLOduration=2.570997673 podStartE2EDuration="2.954065477s" totalImagesPullingTime="383.067804ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:08:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:08:16.953228813 +0000 UTC m=+1273.388131020" watchObservedRunningTime="2026-10-30 17:08:16.954065477 +0000 UTC m=+1273.388967732"
Oct 30 17:08:22.922637 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:22.922598    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw"
Oct 30 17:08:23.624428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:23.624130    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:08:25.037561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:25.037528    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fe3dc03-bnxpc"]
Oct 30 17:08:25.048782 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:25.048749    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fe3dc03-bnxpc"]
Oct 30 17:08:25.699514 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:25.699470    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a0a0b2b-ee78-42b5-9156-a0c5a261fc03" path="/var/lib/kubelet/pods/5a0a0b2b-ee78-42b5-9156-a0c5a261fc03/volumes"
Oct 30 17:08:26.985867 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:08:26.985829    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:08:27.965955 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:27.965918    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"]
Oct 30 17:08:27.972669 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:27.972643    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"
Oct 30 17:08:27.989585 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:27.989553    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7ebeb6e1\"/\"openshift-service-ca.crt\""
Oct 30 17:08:27.995536 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:27.995510    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"]
Oct 30 17:08:28.000886 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:28.000559    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7ebeb6e1\"/\"kube-root-ca.crt\""
Oct 30 17:08:28.053673 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:28.053635    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7ebeb6e1\"/\"default-dockercfg-hqvlt\""
Oct 30 17:08:28.053673 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:28.053636    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7ebeb6e1\"/\"e2e-lifecycle-model-7ebeb6e1-kserve-self-signed-certs\""
Oct 30 17:08:28.094537 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:28.094499    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d32a2aca-cfad-4b04-a149-0e0589dcfa88-home\") pod \"e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg\" (UID: \"d32a2aca-cfad-4b04-a149-0e0589dcfa88\") " pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"
Oct 30 17:08:28.094766 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:28.094620    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d32a2aca-cfad-4b04-a149-0e0589dcfa88-dshm\") pod \"e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg\" (UID: \"d32a2aca-cfad-4b04-a149-0e0589dcfa88\") " pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"
Oct 30 17:08:28.094766 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:28.094680    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d32a2aca-cfad-4b04-a149-0e0589dcfa88-tmp-dir\") pod \"e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg\" (UID: \"d32a2aca-cfad-4b04-a149-0e0589dcfa88\") " pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"
Oct 30 17:08:28.094766 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:28.094731    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwmxz\" (UniqueName: \"kubernetes.io/projected/d32a2aca-cfad-4b04-a149-0e0589dcfa88-kube-api-access-rwmxz\") pod \"e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg\" (UID: \"d32a2aca-cfad-4b04-a149-0e0589dcfa88\") " pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"
Oct 30 17:08:28.094921 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:28.094836    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d32a2aca-cfad-4b04-a149-0e0589dcfa88-tls-certs\") pod \"e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg\" (UID: \"d32a2aca-cfad-4b04-a149-0e0589dcfa88\") " pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"
Oct 30 17:08:28.094921 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:28.094871    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d32a2aca-cfad-4b04-a149-0e0589dcfa88-model-cache\") pod \"e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg\" (UID: \"d32a2aca-cfad-4b04-a149-0e0589dcfa88\") " pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"
Oct 30 17:08:28.196566 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:28.196516    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d32a2aca-cfad-4b04-a149-0e0589dcfa88-home\") pod \"e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg\" (UID: \"d32a2aca-cfad-4b04-a149-0e0589dcfa88\") " pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"
Oct 30 17:08:28.196768 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:28.196605    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d32a2aca-cfad-4b04-a149-0e0589dcfa88-dshm\") pod \"e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg\" (UID: \"d32a2aca-cfad-4b04-a149-0e0589dcfa88\") " pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"
Oct 30 17:08:28.196768 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:28.196661    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d32a2aca-cfad-4b04-a149-0e0589dcfa88-tmp-dir\") pod \"e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg\" (UID: \"d32a2aca-cfad-4b04-a149-0e0589dcfa88\") " pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"
Oct 30 17:08:28.196768 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:28.196722    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwmxz\" (UniqueName: \"kubernetes.io/projected/d32a2aca-cfad-4b04-a149-0e0589dcfa88-kube-api-access-rwmxz\") pod \"e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg\" (UID: \"d32a2aca-cfad-4b04-a149-0e0589dcfa88\") " pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"
Oct 30 17:08:28.196938 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:28.196767    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d32a2aca-cfad-4b04-a149-0e0589dcfa88-tls-certs\") pod \"e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg\" (UID: \"d32a2aca-cfad-4b04-a149-0e0589dcfa88\") " pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"
Oct 30 17:08:28.196938 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:28.196810    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d32a2aca-cfad-4b04-a149-0e0589dcfa88-model-cache\") pod \"e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg\" (UID: \"d32a2aca-cfad-4b04-a149-0e0589dcfa88\") " pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"
Oct 30 17:08:28.197132 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:28.197105    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d32a2aca-cfad-4b04-a149-0e0589dcfa88-tmp-dir\") pod \"e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg\" (UID: \"d32a2aca-cfad-4b04-a149-0e0589dcfa88\") " pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"
Oct 30 17:08:28.197249 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:28.197189    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d32a2aca-cfad-4b04-a149-0e0589dcfa88-home\") pod \"e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg\" (UID: \"d32a2aca-cfad-4b04-a149-0e0589dcfa88\") " pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"
Oct 30 17:08:28.197309 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:28.197272    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d32a2aca-cfad-4b04-a149-0e0589dcfa88-model-cache\") pod \"e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg\" (UID: \"d32a2aca-cfad-4b04-a149-0e0589dcfa88\") " pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"
Oct 30 17:08:28.199192 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:28.199168    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d32a2aca-cfad-4b04-a149-0e0589dcfa88-dshm\") pod \"e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg\" (UID: \"d32a2aca-cfad-4b04-a149-0e0589dcfa88\") " pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"
Oct 30 17:08:28.199793 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:28.199767    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d32a2aca-cfad-4b04-a149-0e0589dcfa88-tls-certs\") pod \"e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg\" (UID: \"d32a2aca-cfad-4b04-a149-0e0589dcfa88\") " pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"
Oct 30 17:08:28.225565 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:28.222274    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwmxz\" (UniqueName: \"kubernetes.io/projected/d32a2aca-cfad-4b04-a149-0e0589dcfa88-kube-api-access-rwmxz\") pod \"e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg\" (UID: \"d32a2aca-cfad-4b04-a149-0e0589dcfa88\") " pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"
Oct 30 17:08:28.292265 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:28.291711    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"
Oct 30 17:08:28.697211 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:28.697186    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"]
Oct 30 17:08:28.698194 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:08:28.698170    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd32a2aca_cfad_4b04_a149_0e0589dcfa88.slice/crio-67b18bc167f391f711967b99af24d3c8dcfa3a24b6cda679d46684cdba055295 WatchSource:0}: Error finding container 67b18bc167f391f711967b99af24d3c8dcfa3a24b6cda679d46684cdba055295: Status 404 returned error can't find the container with id 67b18bc167f391f711967b99af24d3c8dcfa3a24b6cda679d46684cdba055295
Oct 30 17:08:29.015730 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:29.015629    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg" event={"ID":"d32a2aca-cfad-4b04-a149-0e0589dcfa88","Type":"ContainerStarted","Data":"acf77e3620e6321040373b19528930dfaf1aae1c51293c26accf68a9b040a7d8"}
Oct 30 17:08:29.015730 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:29.015685    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg" event={"ID":"d32a2aca-cfad-4b04-a149-0e0589dcfa88","Type":"ContainerStarted","Data":"67b18bc167f391f711967b99af24d3c8dcfa3a24b6cda679d46684cdba055295"}
Oct 30 17:08:29.060526 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:29.060460    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg" podStartSLOduration=2.060441002 podStartE2EDuration="2.060441002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:08:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:08:29.05721554 +0000 UTC m=+1285.492117746" watchObservedRunningTime="2026-10-30 17:08:29.060441002 +0000 UTC m=+1285.495343205"
Oct 30 17:08:29.293150 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:29.293071    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"
Oct 30 17:08:38.293350 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:38.293310    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"
Oct 30 17:08:38.306781 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:38.306748    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"
Oct 30 17:08:39.103038 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:39.103007    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"
Oct 30 17:08:49.041527 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:49.041430    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b718c2b-5zzxh"]
Oct 30 17:08:49.047240 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:49.047208    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b718c2b-5zzxh"]
Oct 30 17:08:49.698327 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:49.698292    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab1552df-b9c5-46fb-880f-ae189e69f391" path="/var/lib/kubelet/pods/ab1552df-b9c5-46fb-880f-ae189e69f391/volumes"
Oct 30 17:08:57.034634 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:08:57.034464    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:08:59.942569 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:08:59.942525    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:09:01.042426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:01.038059    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f603c30-ldbbj"]
Oct 30 17:09:01.052070 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:01.052033    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f603c30-ldbbj"]
Oct 30 17:09:01.632093 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:01.632057    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"]
Oct 30 17:09:01.636492 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:01.636468    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:09:01.651431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:01.650890    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-409387-openshift-default-dockercfg-vj9pn\""
Oct 30 17:09:01.678651 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:01.678605    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"]
Oct 30 17:09:01.698925 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:01.698887    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9587981e-f995-4628-8131-50fcbda9aaf3" path="/var/lib/kubelet/pods/9587981e-f995-4628-8131-50fcbda9aaf3/volumes"
Oct 30 17:09:01.738133 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:01.738089    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c7d819-6442-4e39-b1cc-bb36c62d5732-istio-token\") pod \"e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") " pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:09:01.738337 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:01.738151    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm2hj\" (UniqueName: \"kubernetes.io/projected/83c7d819-6442-4e39-b1cc-bb36c62d5732-kube-api-access-zm2hj\") pod \"e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") " pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:09:01.738664 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:01.738630    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c7d819-6442-4e39-b1cc-bb36c62d5732-istiod-ca-cert\") pod \"e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") " pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:09:01.738804 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:01.738684    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c7d819-6442-4e39-b1cc-bb36c62d5732-istio-data\") pod \"e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") " pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:09:01.738804 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:01.738724    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c7d819-6442-4e39-b1cc-bb36c62d5732-credential-socket\") pod \"e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") " pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:09:01.738804 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:01.738783    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c7d819-6442-4e39-b1cc-bb36c62d5732-istio-podinfo\") pod \"e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") " pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:09:01.740025 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:01.739998    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c7d819-6442-4e39-b1cc-bb36c62d5732-workload-socket\") pod \"e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") " pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:09:01.740235 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:01.740209    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c7d819-6442-4e39-b1cc-bb36c62d5732-workload-certs\") pod \"e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") " pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:09:01.740470 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:01.740368    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c7d819-6442-4e39-b1cc-bb36c62d5732-istio-envoy\") pod \"e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") " pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:09:01.854146 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:01.844466    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c7d819-6442-4e39-b1cc-bb36c62d5732-istiod-ca-cert\") pod \"e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") " pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:09:01.854146 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:01.844573    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c7d819-6442-4e39-b1cc-bb36c62d5732-istio-data\") pod \"e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") " pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:09:01.854146 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:01.844647    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c7d819-6442-4e39-b1cc-bb36c62d5732-credential-socket\") pod \"e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") " pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:09:01.854146 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:01.844723    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c7d819-6442-4e39-b1cc-bb36c62d5732-istio-podinfo\") pod \"e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") " pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:09:01.854146 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:01.844786    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c7d819-6442-4e39-b1cc-bb36c62d5732-workload-socket\") pod \"e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") " pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:09:01.854146 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:01.844823    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c7d819-6442-4e39-b1cc-bb36c62d5732-workload-certs\") pod \"e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") " pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:09:01.854146 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:01.844979    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c7d819-6442-4e39-b1cc-bb36c62d5732-istio-envoy\") pod \"e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") " pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:09:01.854146 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:01.845199    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c7d819-6442-4e39-b1cc-bb36c62d5732-istio-token\") pod \"e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") " pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:09:01.854146 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:01.845252    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm2hj\" (UniqueName: \"kubernetes.io/projected/83c7d819-6442-4e39-b1cc-bb36c62d5732-kube-api-access-zm2hj\") pod \"e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") " pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:09:01.854146 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:01.846166    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c7d819-6442-4e39-b1cc-bb36c62d5732-workload-socket\") pod \"e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") " pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:09:01.854146 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:01.846468    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c7d819-6442-4e39-b1cc-bb36c62d5732-workload-certs\") pod \"e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") " pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:09:01.854146 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:01.849778    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c7d819-6442-4e39-b1cc-bb36c62d5732-istio-data\") pod \"e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") " pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:09:01.854146 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:01.850940    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c7d819-6442-4e39-b1cc-bb36c62d5732-istiod-ca-cert\") pod \"e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") " pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:09:01.854146 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:01.851068    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c7d819-6442-4e39-b1cc-bb36c62d5732-credential-socket\") pod \"e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") " pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:09:01.879426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:01.879357    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c7d819-6442-4e39-b1cc-bb36c62d5732-istio-token\") pod \"e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") " pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:09:01.883470 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:01.882830    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c7d819-6442-4e39-b1cc-bb36c62d5732-istio-envoy\") pod \"e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") " pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:09:01.894284 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:01.894182    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm2hj\" (UniqueName: \"kubernetes.io/projected/83c7d819-6442-4e39-b1cc-bb36c62d5732-kube-api-access-zm2hj\") pod \"e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") " pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:09:01.903427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:01.901891    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c7d819-6442-4e39-b1cc-bb36c62d5732-istio-podinfo\") pod \"e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") " pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:09:01.955089 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:01.955052    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:09:02.418447 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:02.417583    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"]
Oct 30 17:09:02.453803 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:02.453760    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:09:02.454595 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:02.454565    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:09:02.454986 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:02.454797    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:09:03.513201 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:03.513162    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp" event={"ID":"83c7d819-6442-4e39-b1cc-bb36c62d5732","Type":"ContainerStarted","Data":"078e67a901fc5950d2f84bcfdc4a6629a0a9cd2d9f1223d78bb1ce68f6fd71db"}
Oct 30 17:09:03.513824 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:03.513797    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp" event={"ID":"83c7d819-6442-4e39-b1cc-bb36c62d5732","Type":"ContainerStarted","Data":"62022352f2746aa1eff843a66ee0eb2ada57ec3cdd966990bf672563cfa111d7"}
Oct 30 17:09:03.600168 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:03.598196    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp" podStartSLOduration=2.5981721269999998 podStartE2EDuration="2.598172127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:09:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:09:03.580281318 +0000 UTC m=+1320.015183602" watchObservedRunningTime="2026-10-30 17:09:03.598172127 +0000 UTC m=+1320.033074334"
Oct 30 17:09:03.963451 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:03.957660    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:09:03.986430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:03.983791    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:09:04.537430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:04.537326    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:09:04.551133 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:04.547861    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:09:04.776430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:04.775504    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 17:09:06.083430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:06.082484    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm" podUID="bf43bf3d-f126-40a1-a24f-10214a85ceb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 30 17:09:06.102479 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:06.102433    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg" podUID="d32a2aca-cfad-4b04-a149-0e0589dcfa88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Oct 30 17:09:06.615443 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:06.610234    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" podUID="458eba2e-eb03-452a-bcf4-a5e91da83acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:09:06.743533 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:06.733504    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:09:07.102939 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:07.102894    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg" podUID="d32a2aca-cfad-4b04-a149-0e0589dcfa88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:09:07.104663 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:07.104634    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"
Oct 30 17:09:07.156514 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:07.156477    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"
Oct 30 17:09:07.393457 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:07.392649    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:09:09.030751 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:09.030705    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c5e907b-8p2sm"]
Oct 30 17:09:09.042393 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:09.042368    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c5e907b-8p2sm"]
Oct 30 17:09:09.697390 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:09.697354    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="548f3d48-17cc-4345-82e2-2a1babdeccc5" path="/var/lib/kubelet/pods/548f3d48-17cc-4345-82e2-2a1babdeccc5/volumes"
Oct 30 17:09:13.845664 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:13.845632    4433 generic.go:356] "Generic (PLEG): container finished" podID="e4f0b0a7-7c36-4ad7-8573-582828ad701b" containerID="2c1e5d2ddff5a0807279bf736c587e972339e07ddfeb32c620483e4d929ffe24" exitCode=0
Oct 30 17:09:13.846122 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:13.845708    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb" event={"ID":"e4f0b0a7-7c36-4ad7-8573-582828ad701b","Type":"ContainerDied","Data":"2c1e5d2ddff5a0807279bf736c587e972339e07ddfeb32c620483e4d929ffe24"}
Oct 30 17:09:13.846250 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:13.846234    4433 scope.go:122] "RemoveContainer" containerID="2c1e5d2ddff5a0807279bf736c587e972339e07ddfeb32c620483e4d929ffe24"
Oct 30 17:09:13.848210 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:13.848179    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:09:13.848309 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:13.848268    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:09:13.848352 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:13.848307    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:09:14.878382 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:14.878341    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb" event={"ID":"e4f0b0a7-7c36-4ad7-8573-582828ad701b","Type":"ContainerStarted","Data":"4f88f5602331c906a35bb13353dbf6a9a11b9fc3ee8ca48e3da5a76ba30ddd3f"}
Oct 30 17:09:15.142155 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:15.142057    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:09:15.143501 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:15.143470    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 17:09:15.143633 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:15.143534    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb" podUID="e4f0b0a7-7c36-4ad7-8573-582828ad701b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 17:09:16.188863 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:16.188774    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 17:09:16.188863 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:16.188833    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb" podUID="e4f0b0a7-7c36-4ad7-8573-582828ad701b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 17:09:17.141608 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:17.141571    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 17:09:17.141801 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:17.141635    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb" podUID="e4f0b0a7-7c36-4ad7-8573-582828ad701b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 17:09:17.806961 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:17.806917    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:09:18.141652 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:18.141571    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 17:09:18.141652 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:18.141629    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb" podUID="e4f0b0a7-7c36-4ad7-8573-582828ad701b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 17:09:19.141764 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:19.141730    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 17:09:19.142206 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:19.141784    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb" podUID="e4f0b0a7-7c36-4ad7-8573-582828ad701b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 17:09:20.141478 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:20.141437    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 17:09:20.141691 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:20.141521    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb" podUID="e4f0b0a7-7c36-4ad7-8573-582828ad701b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 17:09:21.146965 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:21.146931    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:09:21.148329 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:21.148309    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:09:22.032875 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:22.032843    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-470be499-sw9hh"]
Oct 30 17:09:22.043606 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:22.043580    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-470be499-sw9hh"]
Oct 30 17:09:23.697834 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:23.697796    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaf2e083-ca03-4482-8bc1-97d50fc52dec" path="/var/lib/kubelet/pods/aaf2e083-ca03-4482-8bc1-97d50fc52dec/volumes"
Oct 30 17:09:26.979445 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:09:26.979389    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:09:30.035594 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:30.035548    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-295e55e4-8vsvr"]
Oct 30 17:09:30.046626 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:30.046594    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-295e55e4-8vsvr"]
Oct 30 17:09:31.711198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:31.711139    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea2c222c-a285-4a20-b69a-8c5afd89edfc" path="/var/lib/kubelet/pods/ea2c222c-a285-4a20-b69a-8c5afd89edfc/volumes"
Oct 30 17:09:35.177494 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:35.177460    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-409387-8586759869-rvkdt"]
Oct 30 17:09:35.182695 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:35.182677    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409387-8586759869-rvkdt"
Oct 30 17:09:35.209261 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:35.209228    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-409387-serving-cert\""
Oct 30 17:09:35.215983 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:35.215947    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-409387-8586759869-rvkdt"]
Oct 30 17:09:35.350949 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:35.350908    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59fb466-c9be-42d1-8980-8cf2b28d71be-openshift-service-ca\") pod \"maas-api-e2e-api-a-409387-8586759869-rvkdt\" (UID: \"a59fb466-c9be-42d1-8980-8cf2b28d71be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409387-8586759869-rvkdt"
Oct 30 17:09:35.350949 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:35.350950    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a59fb466-c9be-42d1-8980-8cf2b28d71be-metrics-certs\") pod \"maas-api-e2e-api-a-409387-8586759869-rvkdt\" (UID: \"a59fb466-c9be-42d1-8980-8cf2b28d71be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409387-8586759869-rvkdt"
Oct 30 17:09:35.351195 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:35.351168    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhh95\" (UniqueName: \"kubernetes.io/projected/a59fb466-c9be-42d1-8980-8cf2b28d71be-kube-api-access-zhh95\") pod \"maas-api-e2e-api-a-409387-8586759869-rvkdt\" (UID: \"a59fb466-c9be-42d1-8980-8cf2b28d71be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409387-8586759869-rvkdt"
Oct 30 17:09:35.351361 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:35.351339    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a59fb466-c9be-42d1-8980-8cf2b28d71be-maas-api-tls\") pod \"maas-api-e2e-api-a-409387-8586759869-rvkdt\" (UID: \"a59fb466-c9be-42d1-8980-8cf2b28d71be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409387-8586759869-rvkdt"
Oct 30 17:09:35.453479 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:35.453336    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhh95\" (UniqueName: \"kubernetes.io/projected/a59fb466-c9be-42d1-8980-8cf2b28d71be-kube-api-access-zhh95\") pod \"maas-api-e2e-api-a-409387-8586759869-rvkdt\" (UID: \"a59fb466-c9be-42d1-8980-8cf2b28d71be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409387-8586759869-rvkdt"
Oct 30 17:09:35.453651 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:35.453505    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a59fb466-c9be-42d1-8980-8cf2b28d71be-maas-api-tls\") pod \"maas-api-e2e-api-a-409387-8586759869-rvkdt\" (UID: \"a59fb466-c9be-42d1-8980-8cf2b28d71be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409387-8586759869-rvkdt"
Oct 30 17:09:35.453718 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:35.453666    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59fb466-c9be-42d1-8980-8cf2b28d71be-openshift-service-ca\") pod \"maas-api-e2e-api-a-409387-8586759869-rvkdt\" (UID: \"a59fb466-c9be-42d1-8980-8cf2b28d71be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409387-8586759869-rvkdt"
Oct 30 17:09:35.453718 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:35.453695    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a59fb466-c9be-42d1-8980-8cf2b28d71be-metrics-certs\") pod \"maas-api-e2e-api-a-409387-8586759869-rvkdt\" (UID: \"a59fb466-c9be-42d1-8980-8cf2b28d71be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409387-8586759869-rvkdt"
Oct 30 17:09:35.454640 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:35.454609    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59fb466-c9be-42d1-8980-8cf2b28d71be-openshift-service-ca\") pod \"maas-api-e2e-api-a-409387-8586759869-rvkdt\" (UID: \"a59fb466-c9be-42d1-8980-8cf2b28d71be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409387-8586759869-rvkdt"
Oct 30 17:09:35.459043 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:35.458993    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a59fb466-c9be-42d1-8980-8cf2b28d71be-maas-api-tls\") pod \"maas-api-e2e-api-a-409387-8586759869-rvkdt\" (UID: \"a59fb466-c9be-42d1-8980-8cf2b28d71be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409387-8586759869-rvkdt"
Oct 30 17:09:35.459677 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:35.459656    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a59fb466-c9be-42d1-8980-8cf2b28d71be-metrics-certs\") pod \"maas-api-e2e-api-a-409387-8586759869-rvkdt\" (UID: \"a59fb466-c9be-42d1-8980-8cf2b28d71be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409387-8586759869-rvkdt"
Oct 30 17:09:35.488553 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:35.488514    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhh95\" (UniqueName: \"kubernetes.io/projected/a59fb466-c9be-42d1-8980-8cf2b28d71be-kube-api-access-zhh95\") pod \"maas-api-e2e-api-a-409387-8586759869-rvkdt\" (UID: \"a59fb466-c9be-42d1-8980-8cf2b28d71be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409387-8586759869-rvkdt"
Oct 30 17:09:35.496824 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:35.496794    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409387-8586759869-rvkdt"
Oct 30 17:09:35.830146 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:35.830113    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-409387-8586759869-rvkdt"]
Oct 30 17:09:35.831662 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:09:35.831628    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda59fb466_c9be_42d1_8980_8cf2b28d71be.slice/crio-a3387ef071a6dd030ab40c60c97d6100992de6bc566362c8bbee54ea5f1182cd WatchSource:0}: Error finding container a3387ef071a6dd030ab40c60c97d6100992de6bc566362c8bbee54ea5f1182cd: Status 404 returned error can't find the container with id a3387ef071a6dd030ab40c60c97d6100992de6bc566362c8bbee54ea5f1182cd
Oct 30 17:09:36.029486 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:36.029442    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409387-8586759869-rvkdt" event={"ID":"a59fb466-c9be-42d1-8980-8cf2b28d71be","Type":"ContainerStarted","Data":"a3387ef071a6dd030ab40c60c97d6100992de6bc566362c8bbee54ea5f1182cd"}
Oct 30 17:09:37.038479 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:37.038439    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409387-8586759869-rvkdt" event={"ID":"a59fb466-c9be-42d1-8980-8cf2b28d71be","Type":"ContainerStarted","Data":"328c5f001fa446f26df5e463300863a0ad20f4e8aaf281ad59fbaeae6bcef709"}
Oct 30 17:09:37.038882 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:37.038651    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409387-8586759869-rvkdt"
Oct 30 17:09:37.080550 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:37.080486    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409387-8586759869-rvkdt" podStartSLOduration=1.7404653209999998 podStartE2EDuration="2.080468329s" totalImagesPullingTime="340.003008ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:09:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:09:37.078917652 +0000 UTC m=+1353.513819858" watchObservedRunningTime="2026-10-30 17:09:37.080468329 +0000 UTC m=+1353.515370539"
Oct 30 17:09:42.076823 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:42.076790    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk"]
Oct 30 17:09:42.081388 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:42.081360    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk"
Oct 30 17:09:42.113112 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:42.113067    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk"]
Oct 30 17:09:42.136100 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:42.136046    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-728ad7-serving-cert\""
Oct 30 17:09:42.172042 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:42.172011    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae73bf58-d907-419a-b0c5-6d7b1b43e3f0-metrics-certs\") pod \"maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk\" (UID: \"ae73bf58-d907-419a-b0c5-6d7b1b43e3f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk"
Oct 30 17:09:42.172042 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:42.172055    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae73bf58-d907-419a-b0c5-6d7b1b43e3f0-maas-api-tls\") pod \"maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk\" (UID: \"ae73bf58-d907-419a-b0c5-6d7b1b43e3f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk"
Oct 30 17:09:42.172306 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:42.172209    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28kcj\" (UniqueName: \"kubernetes.io/projected/ae73bf58-d907-419a-b0c5-6d7b1b43e3f0-kube-api-access-28kcj\") pod \"maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk\" (UID: \"ae73bf58-d907-419a-b0c5-6d7b1b43e3f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk"
Oct 30 17:09:42.172353 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:42.172323    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae73bf58-d907-419a-b0c5-6d7b1b43e3f0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk\" (UID: \"ae73bf58-d907-419a-b0c5-6d7b1b43e3f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk"
Oct 30 17:09:42.274095 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:42.274051    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae73bf58-d907-419a-b0c5-6d7b1b43e3f0-maas-api-tls\") pod \"maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk\" (UID: \"ae73bf58-d907-419a-b0c5-6d7b1b43e3f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk"
Oct 30 17:09:42.274511 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:42.274480    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28kcj\" (UniqueName: \"kubernetes.io/projected/ae73bf58-d907-419a-b0c5-6d7b1b43e3f0-kube-api-access-28kcj\") pod \"maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk\" (UID: \"ae73bf58-d907-419a-b0c5-6d7b1b43e3f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk"
Oct 30 17:09:42.274809 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:42.274783    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae73bf58-d907-419a-b0c5-6d7b1b43e3f0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk\" (UID: \"ae73bf58-d907-419a-b0c5-6d7b1b43e3f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk"
Oct 30 17:09:42.275213 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:42.275191    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae73bf58-d907-419a-b0c5-6d7b1b43e3f0-metrics-certs\") pod \"maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk\" (UID: \"ae73bf58-d907-419a-b0c5-6d7b1b43e3f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk"
Oct 30 17:09:42.277206 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:42.277182    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae73bf58-d907-419a-b0c5-6d7b1b43e3f0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk\" (UID: \"ae73bf58-d907-419a-b0c5-6d7b1b43e3f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk"
Oct 30 17:09:42.296039 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:42.294712    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae73bf58-d907-419a-b0c5-6d7b1b43e3f0-metrics-certs\") pod \"maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk\" (UID: \"ae73bf58-d907-419a-b0c5-6d7b1b43e3f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk"
Oct 30 17:09:42.296499 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:42.296439    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae73bf58-d907-419a-b0c5-6d7b1b43e3f0-maas-api-tls\") pod \"maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk\" (UID: \"ae73bf58-d907-419a-b0c5-6d7b1b43e3f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk"
Oct 30 17:09:42.308348 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:42.308278    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28kcj\" (UniqueName: \"kubernetes.io/projected/ae73bf58-d907-419a-b0c5-6d7b1b43e3f0-kube-api-access-28kcj\") pod \"maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk\" (UID: \"ae73bf58-d907-419a-b0c5-6d7b1b43e3f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk"
Oct 30 17:09:42.396607 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:42.396514    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk"
Oct 30 17:09:42.717999 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:42.717969    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk"]
Oct 30 17:09:42.718535 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:09:42.718509    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae73bf58_d907_419a_b0c5_6d7b1b43e3f0.slice/crio-7f69f49a4ba87314bf66d1104b7c21cd8079d813252abb749ddf18015b2dba61 WatchSource:0}: Error finding container 7f69f49a4ba87314bf66d1104b7c21cd8079d813252abb749ddf18015b2dba61: Status 404 returned error can't find the container with id 7f69f49a4ba87314bf66d1104b7c21cd8079d813252abb749ddf18015b2dba61
Oct 30 17:09:43.051470 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:43.051376    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409387-8586759869-rvkdt"
Oct 30 17:09:43.081391 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:43.081353    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk" event={"ID":"ae73bf58-d907-419a-b0c5-6d7b1b43e3f0","Type":"ContainerStarted","Data":"7f69f49a4ba87314bf66d1104b7c21cd8079d813252abb749ddf18015b2dba61"}
Oct 30 17:09:44.035664 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:44.035624    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f284e553-qvdvd"]
Oct 30 17:09:44.046694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:44.046666    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f284e553-qvdvd"]
Oct 30 17:09:44.090309 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:44.090271    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk" event={"ID":"ae73bf58-d907-419a-b0c5-6d7b1b43e3f0","Type":"ContainerStarted","Data":"3e24937ccc5360db4e1c473abb90b1dbd1a221746a6a3ed8345b1d2f0f93353f"}
Oct 30 17:09:44.090815 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:44.090465    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk"
Oct 30 17:09:44.133931 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:44.133867    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk" podStartSLOduration=1.763878306 podStartE2EDuration="2.133848052s" totalImagesPullingTime="369.969746ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:09:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:09:44.131982266 +0000 UTC m=+1360.566884472" watchObservedRunningTime="2026-10-30 17:09:44.133848052 +0000 UTC m=+1360.568750257"
Oct 30 17:09:45.696872 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:45.696837    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70440f57-cc90-47c9-8845-b69d35c82cbc" path="/var/lib/kubelet/pods/70440f57-cc90-47c9-8845-b69d35c82cbc/volumes"
Oct 30 17:09:48.032240 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:48.032205    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99ada05d-hhmhw"]
Oct 30 17:09:48.043591 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:48.043562    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99ada05d-hhmhw"]
Oct 30 17:09:49.698768 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:49.698731    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7afb8ffd-67e2-40e0-89e3-d2cb40667682" path="/var/lib/kubelet/pods/7afb8ffd-67e2-40e0-89e3-d2cb40667682/volumes"
Oct 30 17:09:50.103582 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:50.103500    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk"
Oct 30 17:09:52.407591 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:52.407547    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:09:53.038033 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:53.037997    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2e94962-v57p5"]
Oct 30 17:09:53.041543 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:53.041520    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2e94962-v57p5"]
Oct 30 17:09:53.754427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:09:53.740172    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80030931-275e-44e9-8f11-702f7c863213" path="/var/lib/kubelet/pods/80030931-275e-44e9-8f11-702f7c863213/volumes"
Oct 30 17:09:57.007569 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:09:57.007520    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:10:04.325251 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:04.325221    4433 scope.go:122] "RemoveContainer" containerID="8ec5a69429bcf65692c50fb99431daf08c1b6f6ab6d6866cb43f51b3d7469f18"
Oct 30 17:10:04.354443 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:04.354418    4433 scope.go:122] "RemoveContainer" containerID="aefbee01a0d85052512d2db507a4de61e8060b1a7fdcd02d26c8b4be39178f24"
Oct 30 17:10:04.366150 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:04.366127    4433 scope.go:122] "RemoveContainer" containerID="56414be6672f899945b39cf7781305f5eb09601e16bac907560a6d199bb387d1"
Oct 30 17:10:19.473244 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:19.473205    4433 generic.go:356] "Generic (PLEG): container finished" podID="83c7d819-6442-4e39-b1cc-bb36c62d5732" containerID="078e67a901fc5950d2f84bcfdc4a6629a0a9cd2d9f1223d78bb1ce68f6fd71db" exitCode=0
Oct 30 17:10:19.473687 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:19.473275    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp" event={"ID":"83c7d819-6442-4e39-b1cc-bb36c62d5732","Type":"ContainerDied","Data":"078e67a901fc5950d2f84bcfdc4a6629a0a9cd2d9f1223d78bb1ce68f6fd71db"}
Oct 30 17:10:19.473773 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:19.473757    4433 scope.go:122] "RemoveContainer" containerID="078e67a901fc5950d2f84bcfdc4a6629a0a9cd2d9f1223d78bb1ce68f6fd71db"
Oct 30 17:10:19.475806 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:19.475775    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:10:19.475913 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:19.475866    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:10:19.475960 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:19.475929    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:10:19.537387 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:19.537358    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:10:20.495391 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:20.495353    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp" event={"ID":"83c7d819-6442-4e39-b1cc-bb36c62d5732","Type":"ContainerStarted","Data":"5159b83fa70a8a76b0935192adfcdbe5f84563c87e1b567b4cf980aa3b822f81"}
Oct 30 17:10:20.525264 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:20.525226    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c4c2e4-7594c7fb59-nnq5t"]
Oct 30 17:10:20.528549 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:20.528524    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c2e4-7594c7fb59-nnq5t"
Oct 30 17:10:20.544162 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:20.544135    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c4c2e4\""
Oct 30 17:10:20.544370 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:20.544174    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c4c2e4-dockercfg-l7vzs\""
Oct 30 17:10:20.550305 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:20.550276    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c4c2e4-7594c7fb59-nnq5t"]
Oct 30 17:10:20.638759 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:20.633088    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56a5d068-44be-4afa-8811-b9b89f5dd25e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c4c2e4-7594c7fb59-nnq5t\" (UID: \"56a5d068-44be-4afa-8811-b9b89f5dd25e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c2e4-7594c7fb59-nnq5t"
Oct 30 17:10:20.638759 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:20.633304    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnzk2\" (UniqueName: \"kubernetes.io/projected/56a5d068-44be-4afa-8811-b9b89f5dd25e-kube-api-access-lnzk2\") pod \"payload-pre-processing-e2e-api-b-c4c2e4-7594c7fb59-nnq5t\" (UID: \"56a5d068-44be-4afa-8811-b9b89f5dd25e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c2e4-7594c7fb59-nnq5t"
Oct 30 17:10:20.739425 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:20.736252    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56a5d068-44be-4afa-8811-b9b89f5dd25e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c4c2e4-7594c7fb59-nnq5t\" (UID: \"56a5d068-44be-4afa-8811-b9b89f5dd25e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c2e4-7594c7fb59-nnq5t"
Oct 30 17:10:20.769535 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:20.743558    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnzk2\" (UniqueName: \"kubernetes.io/projected/56a5d068-44be-4afa-8811-b9b89f5dd25e-kube-api-access-lnzk2\") pod \"payload-pre-processing-e2e-api-b-c4c2e4-7594c7fb59-nnq5t\" (UID: \"56a5d068-44be-4afa-8811-b9b89f5dd25e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c2e4-7594c7fb59-nnq5t"
Oct 30 17:10:20.831427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:20.741433    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56a5d068-44be-4afa-8811-b9b89f5dd25e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c4c2e4-7594c7fb59-nnq5t\" (UID: \"56a5d068-44be-4afa-8811-b9b89f5dd25e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c2e4-7594c7fb59-nnq5t"
Oct 30 17:10:20.927328 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:20.927253    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnzk2\" (UniqueName: \"kubernetes.io/projected/56a5d068-44be-4afa-8811-b9b89f5dd25e-kube-api-access-lnzk2\") pod \"payload-pre-processing-e2e-api-b-c4c2e4-7594c7fb59-nnq5t\" (UID: \"56a5d068-44be-4afa-8811-b9b89f5dd25e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c2e4-7594c7fb59-nnq5t"
Oct 30 17:10:20.993426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:20.985186    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:10:21.030482 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.024237    4433 patch_prober.go:29] interesting pod/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 17:10:21.030482 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.024302    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp" podUID="83c7d819-6442-4e39-b1cc-bb36c62d5732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 17:10:21.055264 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.055062    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-873a1264-xlgxz"]
Oct 30 17:10:21.109184 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.109153    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-873a1264-xlgxz"]
Oct 30 17:10:21.160343 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.160306    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c2e4-7594c7fb59-nnq5t"
Oct 30 17:10:21.298745 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.296643    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"]
Oct 30 17:10:21.301396 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.301368    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:10:21.322929 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.321235    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-39fecf-openshift-default-dockercfg-8ldpr\""
Oct 30 17:10:21.326295 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.326264    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"]
Oct 30 17:10:21.434573 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.434532    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cb53e81-fab8-43ff-98c2-e2f91d466226-istio-podinfo\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:10:21.434759 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.434584    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb53e81-fab8-43ff-98c2-e2f91d466226-credential-socket\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:10:21.434759 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.434630    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cb53e81-fab8-43ff-98c2-e2f91d466226-istio-envoy\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:10:21.434759 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.434725    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cb53e81-fab8-43ff-98c2-e2f91d466226-workload-certs\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:10:21.434946 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.434890    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cb53e81-fab8-43ff-98c2-e2f91d466226-istio-token\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:10:21.435005 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.434972    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw9xx\" (UniqueName: \"kubernetes.io/projected/2cb53e81-fab8-43ff-98c2-e2f91d466226-kube-api-access-rw9xx\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:10:21.435061 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.435014    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cb53e81-fab8-43ff-98c2-e2f91d466226-istiod-ca-cert\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:10:21.435061 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.435054    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cb53e81-fab8-43ff-98c2-e2f91d466226-istio-data\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:10:21.435169 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.435134    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb53e81-fab8-43ff-98c2-e2f91d466226-workload-socket\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:10:21.536173 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.536132    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb53e81-fab8-43ff-98c2-e2f91d466226-workload-socket\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:10:21.536687 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.536245    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cb53e81-fab8-43ff-98c2-e2f91d466226-istio-podinfo\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:10:21.536687 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.536276    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb53e81-fab8-43ff-98c2-e2f91d466226-credential-socket\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:10:21.536687 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.536320    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cb53e81-fab8-43ff-98c2-e2f91d466226-istio-envoy\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:10:21.536687 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.536358    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cb53e81-fab8-43ff-98c2-e2f91d466226-workload-certs\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:10:21.536687 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.536548    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cb53e81-fab8-43ff-98c2-e2f91d466226-istio-token\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:10:21.536687 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.536590    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb53e81-fab8-43ff-98c2-e2f91d466226-workload-socket\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:10:21.536687 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.536639    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb53e81-fab8-43ff-98c2-e2f91d466226-credential-socket\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:10:21.536687 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.536648    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw9xx\" (UniqueName: \"kubernetes.io/projected/2cb53e81-fab8-43ff-98c2-e2f91d466226-kube-api-access-rw9xx\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:10:21.537142 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.536699    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cb53e81-fab8-43ff-98c2-e2f91d466226-istiod-ca-cert\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:10:21.537142 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.536750    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cb53e81-fab8-43ff-98c2-e2f91d466226-istio-data\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:10:21.537142 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.536795    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cb53e81-fab8-43ff-98c2-e2f91d466226-workload-certs\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:10:21.537142 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.537028    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cb53e81-fab8-43ff-98c2-e2f91d466226-istio-data\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:10:21.537481 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.537453    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cb53e81-fab8-43ff-98c2-e2f91d466226-istiod-ca-cert\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:10:21.539431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.539381    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cb53e81-fab8-43ff-98c2-e2f91d466226-istio-envoy\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:10:21.539573 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.539556    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cb53e81-fab8-43ff-98c2-e2f91d466226-istio-podinfo\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:10:21.622473 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.614946    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw9xx\" (UniqueName: \"kubernetes.io/projected/2cb53e81-fab8-43ff-98c2-e2f91d466226-kube-api-access-rw9xx\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:10:21.650834 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.637319    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cb53e81-fab8-43ff-98c2-e2f91d466226-istio-token\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:10:21.650834 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.638909    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c4c2e4-7594c7fb59-nnq5t"]
Oct 30 17:10:21.812429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.797626    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f15feeeb-5e97-4e0e-935a-553affd91bb1" path="/var/lib/kubelet/pods/f15feeeb-5e97-4e0e-935a-553affd91bb1/volumes"
Oct 30 17:10:21.946447 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.944245    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:10:21.968589 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.957180    4433 patch_prober.go:29] interesting pod/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 17:10:21.968589 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:21.957378    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp" podUID="83c7d819-6442-4e39-b1cc-bb36c62d5732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 17:10:22.615162 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:22.615056    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"]
Oct 30 17:10:22.644493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:22.644445    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:10:22.644774 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:22.644747    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:10:22.644991 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:22.644954    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:10:22.646195 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:22.645801    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c2e4-7594c7fb59-nnq5t" event={"ID":"56a5d068-44be-4afa-8811-b9b89f5dd25e","Type":"ContainerStarted","Data":"cf88b2b763b3a5e2876c9e16719e87c85a231c40e9f1eb5101cef3c205530370"}
Oct 30 17:10:22.646195 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:22.645857    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c2e4-7594c7fb59-nnq5t" event={"ID":"56a5d068-44be-4afa-8811-b9b89f5dd25e","Type":"ContainerStarted","Data":"cc2da9b18280fc0f603a1710f91cb9880af98d7e2a9b0e2d34774a896d83865e"}
Oct 30 17:10:22.646195 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:22.645900    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c2e4-7594c7fb59-nnq5t"
Oct 30 17:10:22.739229 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:22.739157    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c2e4-7594c7fb59-nnq5t" podStartSLOduration=2.73913706 podStartE2EDuration="2.73913706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:22.729653584 +0000 UTC m=+1399.164555790" watchObservedRunningTime="2026-10-30 17:10:22.73913706 +0000 UTC m=+1399.174039267"
Oct 30 17:10:23.011425 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:23.010673    4433 patch_prober.go:29] interesting pod/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 17:10:23.011425 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:23.010750    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp" podUID="83c7d819-6442-4e39-b1cc-bb36c62d5732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 17:10:23.666431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:23.662783    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt" event={"ID":"2cb53e81-fab8-43ff-98c2-e2f91d466226","Type":"ContainerStarted","Data":"6d4602c51365ec90f03c961d88ff94cb0622649f4408d7387becadfe805181f4"}
Oct 30 17:10:23.666431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:23.662846    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt" event={"ID":"2cb53e81-fab8-43ff-98c2-e2f91d466226","Type":"ContainerStarted","Data":"62bdc9d34bfdb63e6f9bad12ce88794f6e5f531c552376056e10166824d1e298"}
Oct 30 17:10:23.716259 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:23.716180    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt" podStartSLOduration=2.716161806 podStartE2EDuration="2.716161806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:23.712066955 +0000 UTC m=+1400.146969162" watchObservedRunningTime="2026-10-30 17:10:23.716161806 +0000 UTC m=+1400.151064013"
Oct 30 17:10:23.957426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:23.948722    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:10:23.969506 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:23.968618    4433 patch_prober.go:29] interesting pod/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 17:10:23.969506 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:23.968695    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp" podUID="83c7d819-6442-4e39-b1cc-bb36c62d5732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 17:10:23.987669 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:23.986719    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:10:24.702939 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:24.702903    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:10:24.704457 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:24.704434    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:10:24.972488 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:24.972000    4433 patch_prober.go:29] interesting pod/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 17:10:24.972677 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:24.972091    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp" podUID="83c7d819-6442-4e39-b1cc-bb36c62d5732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 17:10:26.107498 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:26.107460    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:10:26.108008 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:26.107934    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:10:26.108927 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:26.108902    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:10:26.993819 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:10:26.993776    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:10:33.664880 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:33.664851    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c2e4-7594c7fb59-nnq5t"
Oct 30 17:10:35.032030 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:35.031992    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f19d8be-9l4rp"]
Oct 30 17:10:35.043018 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:35.042989    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f19d8be-9l4rp"]
Oct 30 17:10:35.680332 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:35.679308    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:10:35.706534 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:35.706497    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f87cc63-37bf-4f29-8605-7c4770c82011" path="/var/lib/kubelet/pods/7f87cc63-37bf-4f29-8605-7c4770c82011/volumes"
Oct 30 17:10:39.503178 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:39.503136    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"]
Oct 30 17:10:39.503831 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:39.503422    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz" podUID="0cb95eff-303b-4088-8ec7-24e9defa3b8a" containerName="istio-proxy" containerID="cri-o://84e8e015d2fdf6a281533d9fc29e1ebbbe095983550b77fac3ebaceadac11209" gracePeriod=30
Oct 30 17:10:44.779871 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:44.779785    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"
Oct 30 17:10:44.866346 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:44.866311    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cb95eff-303b-4088-8ec7-24e9defa3b8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cb95eff-303b-4088-8ec7-24e9defa3b8a-istio-data\") pod \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") "
Oct 30 17:10:44.866561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:44.866443    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cb95eff-303b-4088-8ec7-24e9defa3b8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cb95eff-303b-4088-8ec7-24e9defa3b8a-istiod-ca-cert\") pod \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") "
Oct 30 17:10:44.866561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:44.866507    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cb95eff-303b-4088-8ec7-24e9defa3b8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cb95eff-303b-4088-8ec7-24e9defa3b8a-istio-envoy\") pod \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") "
Oct 30 17:10:44.866708 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:44.866679    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb95eff-303b-4088-8ec7-24e9defa3b8a-istio-data" pod "0cb95eff-303b-4088-8ec7-24e9defa3b8a" (UID: "0cb95eff-303b-4088-8ec7-24e9defa3b8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:10:44.866749 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:44.866691    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cb95eff-303b-4088-8ec7-24e9defa3b8a-kube-api-access-rzhtm\" (UniqueName: \"kubernetes.io/projected/0cb95eff-303b-4088-8ec7-24e9defa3b8a-kube-api-access-rzhtm\") pod \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") "
Oct 30 17:10:44.866789 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:44.866770    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cb95eff-303b-4088-8ec7-24e9defa3b8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cb95eff-303b-4088-8ec7-24e9defa3b8a-workload-certs\") pod \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") "
Oct 30 17:10:44.866867 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:44.866851    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cb95eff-303b-4088-8ec7-24e9defa3b8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb95eff-303b-4088-8ec7-24e9defa3b8a-credential-socket\") pod \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") "
Oct 30 17:10:44.867696 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:44.866952    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cb95eff-303b-4088-8ec7-24e9defa3b8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb95eff-303b-4088-8ec7-24e9defa3b8a-workload-socket\") pod \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") "
Oct 30 17:10:44.867696 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:44.867004    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0cb95eff-303b-4088-8ec7-24e9defa3b8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cb95eff-303b-4088-8ec7-24e9defa3b8a-istio-podinfo\") pod \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") "
Oct 30 17:10:44.867696 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:44.867039    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cb95eff-303b-4088-8ec7-24e9defa3b8a-istio-token\" (UniqueName: \"kubernetes.io/projected/0cb95eff-303b-4088-8ec7-24e9defa3b8a-istio-token\") pod \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\" (UID: \"0cb95eff-303b-4088-8ec7-24e9defa3b8a\") "
Oct 30 17:10:44.867696 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:44.867248    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb95eff-303b-4088-8ec7-24e9defa3b8a-credential-socket" pod "0cb95eff-303b-4088-8ec7-24e9defa3b8a" (UID: "0cb95eff-303b-4088-8ec7-24e9defa3b8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:10:44.867696 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:44.867569    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb95eff-303b-4088-8ec7-24e9defa3b8a-workload-certs" pod "0cb95eff-303b-4088-8ec7-24e9defa3b8a" (UID: "0cb95eff-303b-4088-8ec7-24e9defa3b8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:10:44.867696 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:44.867620    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb95eff-303b-4088-8ec7-24e9defa3b8a-workload-socket" pod "0cb95eff-303b-4088-8ec7-24e9defa3b8a" (UID: "0cb95eff-303b-4088-8ec7-24e9defa3b8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:10:44.868121 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:44.867906    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cb95eff-303b-4088-8ec7-24e9defa3b8a-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:10:44.868121 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:44.867938    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cb95eff-303b-4088-8ec7-24e9defa3b8a-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:10:44.868121 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:44.867952    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb95eff-303b-4088-8ec7-24e9defa3b8a-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:10:44.868121 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:44.867966    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb95eff-303b-4088-8ec7-24e9defa3b8a-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:10:44.868323 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:44.868304    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cb95eff-303b-4088-8ec7-24e9defa3b8a-istiod-ca-cert" pod "0cb95eff-303b-4088-8ec7-24e9defa3b8a" (UID: "0cb95eff-303b-4088-8ec7-24e9defa3b8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:10:44.869544 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:44.869516    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cb95eff-303b-4088-8ec7-24e9defa3b8a-kube-api-access-rzhtm" pod "0cb95eff-303b-4088-8ec7-24e9defa3b8a" (UID: "0cb95eff-303b-4088-8ec7-24e9defa3b8a"). InnerVolumeSpecName "kube-api-access-rzhtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:10:44.869816 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:44.869795    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cb95eff-303b-4088-8ec7-24e9defa3b8a-istio-token" pod "0cb95eff-303b-4088-8ec7-24e9defa3b8a" (UID: "0cb95eff-303b-4088-8ec7-24e9defa3b8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:10:44.870623 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:44.870604    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cb95eff-303b-4088-8ec7-24e9defa3b8a-istio-podinfo" pod "0cb95eff-303b-4088-8ec7-24e9defa3b8a" (UID: "0cb95eff-303b-4088-8ec7-24e9defa3b8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:10:44.870694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:44.870669    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb95eff-303b-4088-8ec7-24e9defa3b8a-istio-envoy" pod "0cb95eff-303b-4088-8ec7-24e9defa3b8a" (UID: "0cb95eff-303b-4088-8ec7-24e9defa3b8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:10:44.969332 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:44.969301    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cb95eff-303b-4088-8ec7-24e9defa3b8a-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:10:44.969332 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:44.969330    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cb95eff-303b-4088-8ec7-24e9defa3b8a-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:10:44.969551 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:44.969343    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzhtm\" (UniqueName: \"kubernetes.io/projected/0cb95eff-303b-4088-8ec7-24e9defa3b8a-kube-api-access-rzhtm\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:10:44.969551 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:44.969355    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cb95eff-303b-4088-8ec7-24e9defa3b8a-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:10:44.969551 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:44.969368    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cb95eff-303b-4088-8ec7-24e9defa3b8a-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:10:45.004982 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:45.004942    4433 generic.go:356] "Generic (PLEG): container finished" podID="0cb95eff-303b-4088-8ec7-24e9defa3b8a" containerID="84e8e015d2fdf6a281533d9fc29e1ebbbe095983550b77fac3ebaceadac11209" exitCode=0
Oct 30 17:10:45.005167 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:45.004994    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"
Oct 30 17:10:45.005167 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:45.005005    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz" event={"ID":"0cb95eff-303b-4088-8ec7-24e9defa3b8a","Type":"ContainerDied","Data":"84e8e015d2fdf6a281533d9fc29e1ebbbe095983550b77fac3ebaceadac11209"}
Oct 30 17:10:45.005167 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:45.005075    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz" event={"ID":"0cb95eff-303b-4088-8ec7-24e9defa3b8a","Type":"ContainerDied","Data":"91a8fef80ced192354f45439b7896496dd1edf3a05c5d81c69ef5ab5d1a8d643"}
Oct 30 17:10:45.005167 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:45.005103    4433 scope.go:122] "RemoveContainer" containerID="84e8e015d2fdf6a281533d9fc29e1ebbbe095983550b77fac3ebaceadac11209"
Oct 30 17:10:45.016897 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:45.016877    4433 scope.go:122] "RemoveContainer" containerID="84e8e015d2fdf6a281533d9fc29e1ebbbe095983550b77fac3ebaceadac11209"
Oct 30 17:10:45.017273 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:10:45.017254    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84e8e015d2fdf6a281533d9fc29e1ebbbe095983550b77fac3ebaceadac11209\": container with ID starting with 84e8e015d2fdf6a281533d9fc29e1ebbbe095983550b77fac3ebaceadac11209 not found: ID does not exist" containerID="84e8e015d2fdf6a281533d9fc29e1ebbbe095983550b77fac3ebaceadac11209"
Oct 30 17:10:45.017345 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:45.017285    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84e8e015d2fdf6a281533d9fc29e1ebbbe095983550b77fac3ebaceadac11209"} err="failed to get container status \"84e8e015d2fdf6a281533d9fc29e1ebbbe095983550b77fac3ebaceadac11209\": rpc error: code = NotFound desc = could not find container \"84e8e015d2fdf6a281533d9fc29e1ebbbe095983550b77fac3ebaceadac11209\": container with ID starting with 84e8e015d2fdf6a281533d9fc29e1ebbbe095983550b77fac3ebaceadac11209 not found: ID does not exist"
Oct 30 17:10:45.057585 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:10:45.057501    4433 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cb95eff_303b_4088_8ec7_24e9defa3b8a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cb95eff_303b_4088_8ec7_24e9defa3b8a.slice/crio-91a8fef80ced192354f45439b7896496dd1edf3a05c5d81c69ef5ab5d1a8d643\": RecentStats: unable to find data in memory cache]"
Oct 30 17:10:45.060227 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:45.060198    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"]
Oct 30 17:10:45.066318 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:45.066293    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-sgmqz"]
Oct 30 17:10:45.698761 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:10:45.698716    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cb95eff-303b-4088-8ec7-24e9defa3b8a" path="/var/lib/kubelet/pods/0cb95eff-303b-4088-8ec7-24e9defa3b8a/volumes"
Oct 30 17:10:56.977843 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:10:56.977807    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:11:01.565283 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:01.565249    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf"]
Oct 30 17:11:01.565954 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:01.565512    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf" podUID="e7761df4-6220-4f57-ac52-ffe62202337d" containerName="maas-api" containerID="cri-o://94b06c7645a6d28cb347af6d6a3a4421c723593943d5b09e8546a31b99aa7c18" gracePeriod=30
Oct 30 17:11:01.731109 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:01.731064    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf" podUID="e7761df4-6220-4f57-ac52-ffe62202337d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.94:8443/health\": dial tcp 10.133.0.94:8443: connect: connection refused"
Oct 30 17:11:02.041908 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.041866    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf"
Oct 30 17:11:02.178427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.175639    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7761df4-6220-4f57-ac52-ffe62202337d-kube-api-access-sr5vq\" (UniqueName: \"kubernetes.io/projected/e7761df4-6220-4f57-ac52-ffe62202337d-kube-api-access-sr5vq\") pod \"e7761df4-6220-4f57-ac52-ffe62202337d\" (UID: \"e7761df4-6220-4f57-ac52-ffe62202337d\") "
Oct 30 17:11:02.178427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.175749    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7761df4-6220-4f57-ac52-ffe62202337d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7761df4-6220-4f57-ac52-ffe62202337d-maas-api-tls\") pod \"e7761df4-6220-4f57-ac52-ffe62202337d\" (UID: \"e7761df4-6220-4f57-ac52-ffe62202337d\") "
Oct 30 17:11:02.178427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.176015    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7761df4-6220-4f57-ac52-ffe62202337d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7761df4-6220-4f57-ac52-ffe62202337d-openshift-service-ca\") pod \"e7761df4-6220-4f57-ac52-ffe62202337d\" (UID: \"e7761df4-6220-4f57-ac52-ffe62202337d\") "
Oct 30 17:11:02.178427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.176112    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7761df4-6220-4f57-ac52-ffe62202337d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7761df4-6220-4f57-ac52-ffe62202337d-metrics-certs\") pod \"e7761df4-6220-4f57-ac52-ffe62202337d\" (UID: \"e7761df4-6220-4f57-ac52-ffe62202337d\") "
Oct 30 17:11:02.181871 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.181175    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7761df4-6220-4f57-ac52-ffe62202337d-openshift-service-ca" pod "e7761df4-6220-4f57-ac52-ffe62202337d" (UID: "e7761df4-6220-4f57-ac52-ffe62202337d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:11:02.183010 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.182978    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7761df4-6220-4f57-ac52-ffe62202337d-metrics-certs" pod "e7761df4-6220-4f57-ac52-ffe62202337d" (UID: "e7761df4-6220-4f57-ac52-ffe62202337d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:11:02.183240 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.183221    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7761df4-6220-4f57-ac52-ffe62202337d-maas-api-tls" pod "e7761df4-6220-4f57-ac52-ffe62202337d" (UID: "e7761df4-6220-4f57-ac52-ffe62202337d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:11:02.183323 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.183275    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7761df4-6220-4f57-ac52-ffe62202337d-kube-api-access-sr5vq" pod "e7761df4-6220-4f57-ac52-ffe62202337d" (UID: "e7761df4-6220-4f57-ac52-ffe62202337d"). InnerVolumeSpecName "kube-api-access-sr5vq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:11:02.260206 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.260171    4433 generic.go:356] "Generic (PLEG): container finished" podID="e7761df4-6220-4f57-ac52-ffe62202337d" containerID="94b06c7645a6d28cb347af6d6a3a4421c723593943d5b09e8546a31b99aa7c18" exitCode=0
Oct 30 17:11:02.260360 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.260217    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf"
Oct 30 17:11:02.260360 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.260255    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf" event={"ID":"e7761df4-6220-4f57-ac52-ffe62202337d","Type":"ContainerDied","Data":"94b06c7645a6d28cb347af6d6a3a4421c723593943d5b09e8546a31b99aa7c18"}
Oct 30 17:11:02.260360 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.260311    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf" event={"ID":"e7761df4-6220-4f57-ac52-ffe62202337d","Type":"ContainerDied","Data":"eb9e35ce4a2733126aad84c78758331166e9a330ba40a33437059c4d8f7a8c4e"}
Oct 30 17:11:02.260360 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.260333    4433 scope.go:122] "RemoveContainer" containerID="94b06c7645a6d28cb347af6d6a3a4421c723593943d5b09e8546a31b99aa7c18"
Oct 30 17:11:02.273586 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.271781    4433 scope.go:122] "RemoveContainer" containerID="94b06c7645a6d28cb347af6d6a3a4421c723593943d5b09e8546a31b99aa7c18"
Oct 30 17:11:02.273586 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:11:02.272254    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94b06c7645a6d28cb347af6d6a3a4421c723593943d5b09e8546a31b99aa7c18\": container with ID starting with 94b06c7645a6d28cb347af6d6a3a4421c723593943d5b09e8546a31b99aa7c18 not found: ID does not exist" containerID="94b06c7645a6d28cb347af6d6a3a4421c723593943d5b09e8546a31b99aa7c18"
Oct 30 17:11:02.273586 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.272318    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94b06c7645a6d28cb347af6d6a3a4421c723593943d5b09e8546a31b99aa7c18"} err="failed to get container status \"94b06c7645a6d28cb347af6d6a3a4421c723593943d5b09e8546a31b99aa7c18\": rpc error: code = NotFound desc = could not find container \"94b06c7645a6d28cb347af6d6a3a4421c723593943d5b09e8546a31b99aa7c18\": container with ID starting with 94b06c7645a6d28cb347af6d6a3a4421c723593943d5b09e8546a31b99aa7c18 not found: ID does not exist"
Oct 30 17:11:02.280660 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.280628    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr5vq\" (UniqueName: \"kubernetes.io/projected/e7761df4-6220-4f57-ac52-ffe62202337d-kube-api-access-sr5vq\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:11:02.280660 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.280662    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7761df4-6220-4f57-ac52-ffe62202337d-maas-api-tls\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:11:02.280870 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.280675    4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7761df4-6220-4f57-ac52-ffe62202337d-openshift-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:11:02.280870 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.280688    4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7761df4-6220-4f57-ac52-ffe62202337d-metrics-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:11:02.456644 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.450677    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf"]
Oct 30 17:11:02.459625 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.459588    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-lnxjf"]
Oct 30 17:11:02.793874 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.793723    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2"]
Oct 30 17:11:02.794783 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.794758    4433 cpu_manager.go:414] "containerMap: removing container" podUID="0cb95eff-303b-4088-8ec7-24e9defa3b8a" containerName="istio-proxy"
Oct 30 17:11:02.794783 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.794783    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cb95eff-303b-4088-8ec7-24e9defa3b8a" containerName="istio-proxy"
Oct 30 17:11:02.796515 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.794817    4433 cpu_manager.go:414] "containerMap: removing container" podUID="e7761df4-6220-4f57-ac52-ffe62202337d" containerName="maas-api"
Oct 30 17:11:02.796515 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.794827    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7761df4-6220-4f57-ac52-ffe62202337d" containerName="maas-api"
Oct 30 17:11:02.796515 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.794984    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7761df4-6220-4f57-ac52-ffe62202337d" containerName="maas-api"
Oct 30 17:11:02.796515 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.795014    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cb95eff-303b-4088-8ec7-24e9defa3b8a" containerName="istio-proxy"
Oct 30 17:11:02.801092 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.801055    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2"
Oct 30 17:11:02.823793 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.823755    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2"]
Oct 30 17:11:02.827114 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.826566    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-39fecf-serving-cert\""
Oct 30 17:11:02.907293 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.907258    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5gqv\" (UniqueName: \"kubernetes.io/projected/b40014e8-60cb-4b74-abbd-efa2eee22ad5-kube-api-access-s5gqv\") pod \"maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2\" (UID: \"b40014e8-60cb-4b74-abbd-efa2eee22ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2"
Oct 30 17:11:02.907503 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.907470    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b40014e8-60cb-4b74-abbd-efa2eee22ad5-maas-api-tls\") pod \"maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2\" (UID: \"b40014e8-60cb-4b74-abbd-efa2eee22ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2"
Oct 30 17:11:02.907623 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.907598    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b40014e8-60cb-4b74-abbd-efa2eee22ad5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2\" (UID: \"b40014e8-60cb-4b74-abbd-efa2eee22ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2"
Oct 30 17:11:02.907689 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:02.907648    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b40014e8-60cb-4b74-abbd-efa2eee22ad5-metrics-certs\") pod \"maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2\" (UID: \"b40014e8-60cb-4b74-abbd-efa2eee22ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2"
Oct 30 17:11:03.009692 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:03.009651    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5gqv\" (UniqueName: \"kubernetes.io/projected/b40014e8-60cb-4b74-abbd-efa2eee22ad5-kube-api-access-s5gqv\") pod \"maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2\" (UID: \"b40014e8-60cb-4b74-abbd-efa2eee22ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2"
Oct 30 17:11:03.009906 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:03.009858    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b40014e8-60cb-4b74-abbd-efa2eee22ad5-maas-api-tls\") pod \"maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2\" (UID: \"b40014e8-60cb-4b74-abbd-efa2eee22ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2"
Oct 30 17:11:03.009982 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:03.009967    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b40014e8-60cb-4b74-abbd-efa2eee22ad5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2\" (UID: \"b40014e8-60cb-4b74-abbd-efa2eee22ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2"
Oct 30 17:11:03.010042 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:03.009993    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b40014e8-60cb-4b74-abbd-efa2eee22ad5-metrics-certs\") pod \"maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2\" (UID: \"b40014e8-60cb-4b74-abbd-efa2eee22ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2"
Oct 30 17:11:03.010102 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:11:03.010058    4433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-39fecf-serving-cert: secret "maas-api-e2e-inf-b-39fecf-serving-cert" not found
Oct 30 17:11:03.010164 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:11:03.010119    4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b40014e8-60cb-4b74-abbd-efa2eee22ad5-maas-api-tls podName:b40014e8-60cb-4b74-abbd-efa2eee22ad5 nodeName:}" failed. No retries permitted until 2026-10-30 17:11:03.510100391 +0000 UTC m=+1439.945002577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b40014e8-60cb-4b74-abbd-efa2eee22ad5-maas-api-tls") pod "maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2" (UID: "b40014e8-60cb-4b74-abbd-efa2eee22ad5") : secret "maas-api-e2e-inf-b-39fecf-serving-cert" not found
Oct 30 17:11:03.011052 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:03.011023    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b40014e8-60cb-4b74-abbd-efa2eee22ad5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2\" (UID: \"b40014e8-60cb-4b74-abbd-efa2eee22ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2"
Oct 30 17:11:03.013041 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:03.013014    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b40014e8-60cb-4b74-abbd-efa2eee22ad5-metrics-certs\") pod \"maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2\" (UID: \"b40014e8-60cb-4b74-abbd-efa2eee22ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2"
Oct 30 17:11:03.044314 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:03.044236    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5gqv\" (UniqueName: \"kubernetes.io/projected/b40014e8-60cb-4b74-abbd-efa2eee22ad5-kube-api-access-s5gqv\") pod \"maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2\" (UID: \"b40014e8-60cb-4b74-abbd-efa2eee22ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2"
Oct 30 17:11:03.523044 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:03.523001    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b40014e8-60cb-4b74-abbd-efa2eee22ad5-maas-api-tls\") pod \"maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2\" (UID: \"b40014e8-60cb-4b74-abbd-efa2eee22ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2"
Oct 30 17:11:03.525635 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:03.525599    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b40014e8-60cb-4b74-abbd-efa2eee22ad5-maas-api-tls\") pod \"maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2\" (UID: \"b40014e8-60cb-4b74-abbd-efa2eee22ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2"
Oct 30 17:11:03.697725 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:03.697694    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7761df4-6220-4f57-ac52-ffe62202337d" path="/var/lib/kubelet/pods/e7761df4-6220-4f57-ac52-ffe62202337d/volumes"
Oct 30 17:11:03.729273 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:03.729239    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2"
Oct 30 17:11:04.072040 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:04.072005    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2"]
Oct 30 17:11:04.080433 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:11:04.080384    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb40014e8_60cb_4b74_abbd_efa2eee22ad5.slice/crio-149d313d8d21ec0da74b3385786fd2b8caac5a25203315cdb5559b4c706829e6 WatchSource:0}: Error finding container 149d313d8d21ec0da74b3385786fd2b8caac5a25203315cdb5559b4c706829e6: Status 404 returned error can't find the container with id 149d313d8d21ec0da74b3385786fd2b8caac5a25203315cdb5559b4c706829e6
Oct 30 17:11:04.287829 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:04.287788    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2" event={"ID":"b40014e8-60cb-4b74-abbd-efa2eee22ad5","Type":"ContainerStarted","Data":"149d313d8d21ec0da74b3385786fd2b8caac5a25203315cdb5559b4c706829e6"}
Oct 30 17:11:04.404606 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:04.404521    4433 scope.go:122] "RemoveContainer" containerID="b60edfa237cec9c672a22af128c6e559584fbe7476ed50046717990a03236737"
Oct 30 17:11:04.414926 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:04.414903    4433 scope.go:122] "RemoveContainer" containerID="138fe3a84b4e56e7c224dec77329a6e7788027414a2ed606fb1b596df70c1039"
Oct 30 17:11:04.441594 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:04.437951    4433 scope.go:122] "RemoveContainer" containerID="2f3fc3ac65342a91e0bae958a1965c53812ec254dc04e3844acda8351308af58"
Oct 30 17:11:04.494353 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:04.492112    4433 scope.go:122] "RemoveContainer" containerID="f445d532af529a451a83433d68ed1bb8e7a1e37e7bc8c6b36977795c14bede07"
Oct 30 17:11:04.515216 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:04.515194    4433 scope.go:122] "RemoveContainer" containerID="c2055ddc37bfaa557ac3bc0f9fc4268c5ebb44fea0b569780a161696ac9c8102"
Oct 30 17:11:04.524942 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:04.524916    4433 scope.go:122] "RemoveContainer" containerID="009c76a50730ac358707b28576f6ed70f5dba7ab2cf007df3151e69b1e52c1ef"
Oct 30 17:11:05.301772 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:05.301730    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2" event={"ID":"b40014e8-60cb-4b74-abbd-efa2eee22ad5","Type":"ContainerStarted","Data":"1e8e7592d9e6bbc6f12d82803290ee0b4cae50e21badf1f1652dd662ba00880e"}
Oct 30 17:11:05.302231 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:05.301919    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2"
Oct 30 17:11:05.345752 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:05.345700    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2" podStartSLOduration=2.912967441 podStartE2EDuration="3.345685337s" totalImagesPullingTime="432.717896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:05.342735532 +0000 UTC m=+1441.777637738" watchObservedRunningTime="2026-10-30 17:11:05.345685337 +0000 UTC m=+1441.780587541"
Oct 30 17:11:11.324936 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:11.324907    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2"
Oct 30 17:11:11.354836 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:11.354805    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-gvpm6"]
Oct 30 17:11:11.359651 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:11.359631    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-gvpm6"
Oct 30 17:11:11.386317 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:11.377315    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7ebeb6e1\""
Oct 30 17:11:11.386317 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:11.383804    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7ebeb6e1-dockercfg-42m7h\""
Oct 30 17:11:11.394384 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:11.394333    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2mbv\" (UniqueName: \"kubernetes.io/projected/5b1905ac-5888-4977-b8df-92959b8c4d67-kube-api-access-g2mbv\") pod \"payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-gvpm6\" (UID: \"5b1905ac-5888-4977-b8df-92959b8c4d67\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-gvpm6"
Oct 30 17:11:11.394754 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:11.394698    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b1905ac-5888-4977-b8df-92959b8c4d67-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-gvpm6\" (UID: \"5b1905ac-5888-4977-b8df-92959b8c4d67\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-gvpm6"
Oct 30 17:11:11.402465 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:11.401610    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-gvpm6"]
Oct 30 17:11:11.526426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:11.514347    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b1905ac-5888-4977-b8df-92959b8c4d67-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-gvpm6\" (UID: \"5b1905ac-5888-4977-b8df-92959b8c4d67\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-gvpm6"
Oct 30 17:11:11.526426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:11.514556    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2mbv\" (UniqueName: \"kubernetes.io/projected/5b1905ac-5888-4977-b8df-92959b8c4d67-kube-api-access-g2mbv\") pod \"payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-gvpm6\" (UID: \"5b1905ac-5888-4977-b8df-92959b8c4d67\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-gvpm6"
Oct 30 17:11:11.526426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:11.515542    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b1905ac-5888-4977-b8df-92959b8c4d67-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-gvpm6\" (UID: \"5b1905ac-5888-4977-b8df-92959b8c4d67\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-gvpm6"
Oct 30 17:11:11.559893 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:11.559841    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2mbv\" (UniqueName: \"kubernetes.io/projected/5b1905ac-5888-4977-b8df-92959b8c4d67-kube-api-access-g2mbv\") pod \"payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-gvpm6\" (UID: \"5b1905ac-5888-4977-b8df-92959b8c4d67\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-gvpm6"
Oct 30 17:11:11.692720 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:11.692684    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-gvpm6"
Oct 30 17:11:12.047602 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:12.047572    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-gvpm6"]
Oct 30 17:11:12.050578 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:11:12.050549    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b1905ac_5888_4977_b8df_92959b8c4d67.slice/crio-3a9bd3bb347fcc30684074143a948bf3ab4a9cd7e01c36add2554165146f4d0c WatchSource:0}: Error finding container 3a9bd3bb347fcc30684074143a948bf3ab4a9cd7e01c36add2554165146f4d0c: Status 404 returned error can't find the container with id 3a9bd3bb347fcc30684074143a948bf3ab4a9cd7e01c36add2554165146f4d0c
Oct 30 17:11:12.303997 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:12.303920    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-432680-569f95cccc-csjrf"]
Oct 30 17:11:12.308324 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:12.308307    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-432680-569f95cccc-csjrf"
Oct 30 17:11:12.326035 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:12.326008    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-432680-serving-cert\""
Oct 30 17:11:12.328527 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:12.328504    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-432680-569f95cccc-csjrf"]
Oct 30 17:11:12.404697 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:12.404660    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-gvpm6" event={"ID":"5b1905ac-5888-4977-b8df-92959b8c4d67","Type":"ContainerStarted","Data":"5c3acc58b0f244707e7b43ea33a809528b82012f29753471eefd5183312c3805"}
Oct 30 17:11:12.404858 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:12.404726    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-gvpm6" event={"ID":"5b1905ac-5888-4977-b8df-92959b8c4d67","Type":"ContainerStarted","Data":"3a9bd3bb347fcc30684074143a948bf3ab4a9cd7e01c36add2554165146f4d0c"}
Oct 30 17:11:12.404858 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:12.404756    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-gvpm6"
Oct 30 17:11:12.445176 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:12.445120    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-gvpm6" podStartSLOduration=1.445104263 podStartE2EDuration="1.445104263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:12.442939211 +0000 UTC m=+1448.877841419" watchObservedRunningTime="2026-10-30 17:11:12.445104263 +0000 UTC m=+1448.880006468"
Oct 30 17:11:12.455940 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:12.455907    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs284\" (UniqueName: \"kubernetes.io/projected/d8c08f00-b142-47ba-a441-6ac129e0f3e8-kube-api-access-cs284\") pod \"maas-api-e2e-api-a-432680-569f95cccc-csjrf\" (UID: \"d8c08f00-b142-47ba-a441-6ac129e0f3e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-432680-569f95cccc-csjrf"
Oct 30 17:11:12.456085 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:12.455968    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c08f00-b142-47ba-a441-6ac129e0f3e8-metrics-certs\") pod \"maas-api-e2e-api-a-432680-569f95cccc-csjrf\" (UID: \"d8c08f00-b142-47ba-a441-6ac129e0f3e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-432680-569f95cccc-csjrf"
Oct 30 17:11:12.456159 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:12.456138    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c08f00-b142-47ba-a441-6ac129e0f3e8-openshift-service-ca\") pod \"maas-api-e2e-api-a-432680-569f95cccc-csjrf\" (UID: \"d8c08f00-b142-47ba-a441-6ac129e0f3e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-432680-569f95cccc-csjrf"
Oct 30 17:11:12.456229 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:12.456211    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c08f00-b142-47ba-a441-6ac129e0f3e8-maas-api-tls\") pod \"maas-api-e2e-api-a-432680-569f95cccc-csjrf\" (UID: \"d8c08f00-b142-47ba-a441-6ac129e0f3e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-432680-569f95cccc-csjrf"
Oct 30 17:11:12.570630 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:12.566888    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c08f00-b142-47ba-a441-6ac129e0f3e8-openshift-service-ca\") pod \"maas-api-e2e-api-a-432680-569f95cccc-csjrf\" (UID: \"d8c08f00-b142-47ba-a441-6ac129e0f3e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-432680-569f95cccc-csjrf"
Oct 30 17:11:12.570630 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:12.567010    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c08f00-b142-47ba-a441-6ac129e0f3e8-maas-api-tls\") pod \"maas-api-e2e-api-a-432680-569f95cccc-csjrf\" (UID: \"d8c08f00-b142-47ba-a441-6ac129e0f3e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-432680-569f95cccc-csjrf"
Oct 30 17:11:12.570630 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:12.567390    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs284\" (UniqueName: \"kubernetes.io/projected/d8c08f00-b142-47ba-a441-6ac129e0f3e8-kube-api-access-cs284\") pod \"maas-api-e2e-api-a-432680-569f95cccc-csjrf\" (UID: \"d8c08f00-b142-47ba-a441-6ac129e0f3e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-432680-569f95cccc-csjrf"
Oct 30 17:11:12.572176 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:12.571983    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c08f00-b142-47ba-a441-6ac129e0f3e8-metrics-certs\") pod \"maas-api-e2e-api-a-432680-569f95cccc-csjrf\" (UID: \"d8c08f00-b142-47ba-a441-6ac129e0f3e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-432680-569f95cccc-csjrf"
Oct 30 17:11:12.573227 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:12.569022    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c08f00-b142-47ba-a441-6ac129e0f3e8-openshift-service-ca\") pod \"maas-api-e2e-api-a-432680-569f95cccc-csjrf\" (UID: \"d8c08f00-b142-47ba-a441-6ac129e0f3e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-432680-569f95cccc-csjrf"
Oct 30 17:11:12.578352 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:12.577569    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c08f00-b142-47ba-a441-6ac129e0f3e8-maas-api-tls\") pod \"maas-api-e2e-api-a-432680-569f95cccc-csjrf\" (UID: \"d8c08f00-b142-47ba-a441-6ac129e0f3e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-432680-569f95cccc-csjrf"
Oct 30 17:11:12.581613 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:12.581541    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c08f00-b142-47ba-a441-6ac129e0f3e8-metrics-certs\") pod \"maas-api-e2e-api-a-432680-569f95cccc-csjrf\" (UID: \"d8c08f00-b142-47ba-a441-6ac129e0f3e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-432680-569f95cccc-csjrf"
Oct 30 17:11:12.601434 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:12.592018    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs284\" (UniqueName: \"kubernetes.io/projected/d8c08f00-b142-47ba-a441-6ac129e0f3e8-kube-api-access-cs284\") pod \"maas-api-e2e-api-a-432680-569f95cccc-csjrf\" (UID: \"d8c08f00-b142-47ba-a441-6ac129e0f3e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-432680-569f95cccc-csjrf"
Oct 30 17:11:12.622497 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:12.622463    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-432680-569f95cccc-csjrf"
Oct 30 17:11:12.938853 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:12.938824    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-432680-569f95cccc-csjrf"]
Oct 30 17:11:12.940520 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:11:12.940483    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c08f00_b142_47ba_a441_6ac129e0f3e8.slice/crio-c0e69bbe337c7e531aa1ba81e7d6654ab5bfa8d547a61a2063df4f5db33185e5 WatchSource:0}: Error finding container c0e69bbe337c7e531aa1ba81e7d6654ab5bfa8d547a61a2063df4f5db33185e5: Status 404 returned error can't find the container with id c0e69bbe337c7e531aa1ba81e7d6654ab5bfa8d547a61a2063df4f5db33185e5
Oct 30 17:11:12.942088 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:12.942069    4433 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 17:11:13.415826 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:13.415768    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-432680-569f95cccc-csjrf" event={"ID":"d8c08f00-b142-47ba-a441-6ac129e0f3e8","Type":"ContainerStarted","Data":"b1e7f5b6d48b73441e575909cdce41f915d5ffd7951a697a97b80a2256f01e34"}
Oct 30 17:11:13.416310 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:13.415844    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-432680-569f95cccc-csjrf" event={"ID":"d8c08f00-b142-47ba-a441-6ac129e0f3e8","Type":"ContainerStarted","Data":"c0e69bbe337c7e531aa1ba81e7d6654ab5bfa8d547a61a2063df4f5db33185e5"}
Oct 30 17:11:13.416448 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:13.416394    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-432680-569f95cccc-csjrf"
Oct 30 17:11:13.459343 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:13.459288    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-432680-569f95cccc-csjrf" podStartSLOduration=1.164738243 podStartE2EDuration="1.459273015s" totalImagesPullingTime="294.534772ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:13.459017422 +0000 UTC m=+1449.893919629" watchObservedRunningTime="2026-10-30 17:11:13.459273015 +0000 UTC m=+1449.894175221"
Oct 30 17:11:19.435569 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:19.435535    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-432680-569f95cccc-csjrf"
Oct 30 17:11:23.419170 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:23.419136    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-gvpm6"
Oct 30 17:11:26.979219 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:11:26.979178    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:11:29.079279 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.079238    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"]
Oct 30 17:11:29.101980 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.096731    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"
Oct 30 17:11:29.116039 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.116012    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c4c2e4-openshift-default-dockercfg-7cm86\""
Oct 30 17:11:29.121651 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.121469    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"]
Oct 30 17:11:29.189394 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.189303    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aab3820c-99d0-43f4-8dc6-f65893969990-istio-token\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"
Oct 30 17:11:29.189849 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.189814    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzn6j\" (UniqueName: \"kubernetes.io/projected/aab3820c-99d0-43f4-8dc6-f65893969990-kube-api-access-pzn6j\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"
Oct 30 17:11:29.189989 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.189922    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aab3820c-99d0-43f4-8dc6-f65893969990-istio-podinfo\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"
Oct 30 17:11:29.190051 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.189994    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aab3820c-99d0-43f4-8dc6-f65893969990-istio-envoy\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"
Oct 30 17:11:29.190108 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.190048    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aab3820c-99d0-43f4-8dc6-f65893969990-workload-certs\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"
Oct 30 17:11:29.190185 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.190160    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab3820c-99d0-43f4-8dc6-f65893969990-workload-socket\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"
Oct 30 17:11:29.190506 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.190477    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aab3820c-99d0-43f4-8dc6-f65893969990-istiod-ca-cert\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"
Oct 30 17:11:29.190635 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.190538    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab3820c-99d0-43f4-8dc6-f65893969990-credential-socket\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"
Oct 30 17:11:29.190753 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.190727    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aab3820c-99d0-43f4-8dc6-f65893969990-istio-data\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"
Oct 30 17:11:29.292711 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.292672    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aab3820c-99d0-43f4-8dc6-f65893969990-istio-data\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"
Oct 30 17:11:29.292896 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.292746    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aab3820c-99d0-43f4-8dc6-f65893969990-istio-token\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"
Oct 30 17:11:29.292995 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.292923    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzn6j\" (UniqueName: \"kubernetes.io/projected/aab3820c-99d0-43f4-8dc6-f65893969990-kube-api-access-pzn6j\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"
Oct 30 17:11:29.292995 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.292961    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aab3820c-99d0-43f4-8dc6-f65893969990-istio-podinfo\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"
Oct 30 17:11:29.293102 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.293014    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aab3820c-99d0-43f4-8dc6-f65893969990-istio-envoy\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"
Oct 30 17:11:29.293102 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.293057    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aab3820c-99d0-43f4-8dc6-f65893969990-workload-certs\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"
Oct 30 17:11:29.293198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.293116    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab3820c-99d0-43f4-8dc6-f65893969990-workload-socket\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"
Oct 30 17:11:29.293340 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.293294    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aab3820c-99d0-43f4-8dc6-f65893969990-istio-data\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"
Oct 30 17:11:29.293340 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.293304    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aab3820c-99d0-43f4-8dc6-f65893969990-istiod-ca-cert\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"
Oct 30 17:11:29.293493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.293390    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab3820c-99d0-43f4-8dc6-f65893969990-credential-socket\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"
Oct 30 17:11:29.293831 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.293809    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab3820c-99d0-43f4-8dc6-f65893969990-credential-socket\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"
Oct 30 17:11:29.294080 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.294056    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab3820c-99d0-43f4-8dc6-f65893969990-workload-socket\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"
Oct 30 17:11:29.294194 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.294060    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aab3820c-99d0-43f4-8dc6-f65893969990-istiod-ca-cert\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"
Oct 30 17:11:29.294522 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.294497    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aab3820c-99d0-43f4-8dc6-f65893969990-workload-certs\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"
Oct 30 17:11:29.296307 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.296282    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aab3820c-99d0-43f4-8dc6-f65893969990-istio-envoy\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"
Oct 30 17:11:29.300162 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.296446    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aab3820c-99d0-43f4-8dc6-f65893969990-istio-podinfo\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"
Oct 30 17:11:29.316337 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.316309    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aab3820c-99d0-43f4-8dc6-f65893969990-istio-token\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"
Oct 30 17:11:29.316497 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.316471    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzn6j\" (UniqueName: \"kubernetes.io/projected/aab3820c-99d0-43f4-8dc6-f65893969990-kube-api-access-pzn6j\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"
Oct 30 17:11:29.430189 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.430139    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"
Oct 30 17:11:29.773498 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.773457    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"]
Oct 30 17:11:29.774781 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:11:29.774751    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaab3820c_99d0_43f4_8dc6_f65893969990.slice/crio-30c7d3002f99a9de2076fb1706872bdbc4f8f0ec810316c7dc53466fd6580e44 WatchSource:0}: Error finding container 30c7d3002f99a9de2076fb1706872bdbc4f8f0ec810316c7dc53466fd6580e44: Status 404 returned error can't find the container with id 30c7d3002f99a9de2076fb1706872bdbc4f8f0ec810316c7dc53466fd6580e44
Oct 30 17:11:29.777966 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.777929    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:11:29.778072 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.778003    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:11:29.778072 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:29.778030    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:11:30.648842 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:30.648807    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf" event={"ID":"aab3820c-99d0-43f4-8dc6-f65893969990","Type":"ContainerStarted","Data":"c4234c955cbd38fa9b30d0010f78d218ba4b852c974192b3a80b70e39f7b0197"}
Oct 30 17:11:30.649353 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:30.648881    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf" event={"ID":"aab3820c-99d0-43f4-8dc6-f65893969990","Type":"ContainerStarted","Data":"30c7d3002f99a9de2076fb1706872bdbc4f8f0ec810316c7dc53466fd6580e44"}
Oct 30 17:11:30.749504 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:30.749389    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf" podStartSLOduration=1.7493701069999998 podStartE2EDuration="1.749370107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:30.738027832 +0000 UTC m=+1467.172930039" watchObservedRunningTime="2026-10-30 17:11:30.749370107 +0000 UTC m=+1467.184272314"
Oct 30 17:11:31.458428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:31.456462    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"
Oct 30 17:11:31.458428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:31.456695    4433 patch_prober.go:29] interesting pod/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 17:11:31.458428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:31.456801    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf" podUID="aab3820c-99d0-43f4-8dc6-f65893969990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 17:11:32.431570 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:32.431535    4433 patch_prober.go:29] interesting pod/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 17:11:32.431986 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:32.431587    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf" podUID="aab3820c-99d0-43f4-8dc6-f65893969990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 17:11:33.430665 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:33.430622    4433 patch_prober.go:29] interesting pod/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 17:11:33.430843 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:33.430682    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf" podUID="aab3820c-99d0-43f4-8dc6-f65893969990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 17:11:34.431439 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:34.431388    4433 patch_prober.go:29] interesting pod/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 17:11:34.431875 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:34.431454    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf" podUID="aab3820c-99d0-43f4-8dc6-f65893969990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 17:11:35.430559 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:35.430516    4433 patch_prober.go:29] interesting pod/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 17:11:35.430762 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:35.430590    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf" podUID="aab3820c-99d0-43f4-8dc6-f65893969990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 17:11:36.655953 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:36.655911    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"
Oct 30 17:11:36.656449 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:36.656386    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"
Oct 30 17:11:36.657475 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:11:36.657453    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"
Oct 30 17:11:56.986977 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:11:56.986895    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:12:03.043524 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:03.043488    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:12:04.861424 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:04.857033    4433 scope.go:122] "RemoveContainer" containerID="b0cfddaf6430b3491e3671650af52dcb32bc7809de1f93fb1e46b7a7392e786f"
Oct 30 17:12:04.951673 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:04.951640    4433 scope.go:122] "RemoveContainer" containerID="1e49f5c71b1694e6eba656d5d36485cce67e267b3640a5c77b8b8c2e4099bb49"
Oct 30 17:12:08.350921 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:08.350868    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 17:12:08.523685 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:08.523636    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:12:08.657875 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:08.657746    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:12:08.732586 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:08.732527    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:12:08.822894 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:08.822839    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 17:12:09.106442 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:09.105503    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:12:09.158552 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:09.158502    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg" podUID="d32a2aca-cfad-4b04-a149-0e0589dcfa88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:12:09.304441 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:09.303543    4433 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg" podUID="d32a2aca-cfad-4b04-a149-0e0589dcfa88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:12:09.373318 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:09.373209    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:12:09.374764 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:09.374739    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:12:09.525545 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:09.525490    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 30 17:12:09.526684 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:09.526653    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 17:12:09.627488 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:09.611294    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" podUID="458eba2e-eb03-452a-bcf4-a5e91da83acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:12:09.658559 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:09.658509    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:12:09.661996 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:09.659545    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 17:12:09.678545 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:09.678514    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 17:12:09.839573 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:09.839540    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 17:12:09.867309 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:09.860945    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:12:09.867309 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:09.861734    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 17:12:10.123220 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:10.123162    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:12:10.150150 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:10.123560    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm" podUID="bf43bf3d-f126-40a1-a24f-10214a85ceb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 30 17:12:10.177484 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:10.177434    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg" podUID="d32a2aca-cfad-4b04-a149-0e0589dcfa88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:12:10.334037 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:10.331723    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 17:12:10.334037 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:10.331788    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"
Oct 30 17:12:10.337586 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:10.337561    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:12:10.338305 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:10.338288    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"
Oct 30 17:12:10.339015 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:10.338990    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:12:10.431641 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:10.405059    4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:12:10.545323 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:10.545233    4433 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 17:12:10.691580 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:10.691481    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:12:10.774462 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:10.750284    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 17:12:10.866983 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:10.866939    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:12:11.006511 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:11.006426    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 17:12:17.600311 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:17.600267    4433 generic.go:356] "Generic (PLEG): container finished" podID="2cb53e81-fab8-43ff-98c2-e2f91d466226" containerID="6d4602c51365ec90f03c961d88ff94cb0622649f4408d7387becadfe805181f4" exitCode=0
Oct 30 17:12:17.600840 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:17.600530    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt" event={"ID":"2cb53e81-fab8-43ff-98c2-e2f91d466226","Type":"ContainerDied","Data":"6d4602c51365ec90f03c961d88ff94cb0622649f4408d7387becadfe805181f4"}
Oct 30 17:12:17.601112 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:17.601091    4433 scope.go:122] "RemoveContainer" containerID="6d4602c51365ec90f03c961d88ff94cb0622649f4408d7387becadfe805181f4"
Oct 30 17:12:17.603674 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:17.603635    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:12:17.603787 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:17.603730    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:12:17.603787 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:17.603778    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:12:18.609998 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:18.609946    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt" event={"ID":"2cb53e81-fab8-43ff-98c2-e2f91d466226","Type":"ContainerStarted","Data":"a0166291b4b1ae17764b9fc4188ecd42a9b84698a59c48564284608e8c8e54ff"}
Oct 30 17:12:18.945473 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:18.945381    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:12:18.946813 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:18.946789    4433 patch_prober.go:29] interesting pod/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 17:12:18.946902 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:18.946847    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt" podUID="2cb53e81-fab8-43ff-98c2-e2f91d466226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 17:12:19.945122 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:19.945085    4433 patch_prober.go:29] interesting pod/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 17:12:19.945592 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:19.945144    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt" podUID="2cb53e81-fab8-43ff-98c2-e2f91d466226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 17:12:20.945679 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:20.945646    4433 patch_prober.go:29] interesting pod/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 17:12:20.946092 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:20.945700    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt" podUID="2cb53e81-fab8-43ff-98c2-e2f91d466226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 17:12:21.951425 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:21.950887    4433 patch_prober.go:29] interesting pod/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 17:12:21.951425 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:21.950958    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt" podUID="2cb53e81-fab8-43ff-98c2-e2f91d466226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 17:12:22.945152 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:22.945111    4433 patch_prober.go:29] interesting pod/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 17:12:22.945361 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:22.945170    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt" podUID="2cb53e81-fab8-43ff-98c2-e2f91d466226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 17:12:23.945539 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:23.945498    4433 patch_prober.go:29] interesting pod/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 17:12:23.946054 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:23.945552    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt" podUID="2cb53e81-fab8-43ff-98c2-e2f91d466226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 17:12:24.703581 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:24.703543    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:12:24.951169 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:24.951131    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:12:25.664684 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:25.664650    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:12:26.980381 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:12:26.980340    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:12:38.109226 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:38.109186    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-gvpm6"]
Oct 30 17:12:38.872334 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:38.872302    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-409387-8586759869-rvkdt"]
Oct 30 17:12:38.872929 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:38.872828    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409387-8586759869-rvkdt" podUID="a59fb466-c9be-42d1-8980-8cf2b28d71be" containerName="maas-api" containerID="cri-o://328c5f001fa446f26df5e463300863a0ad20f4e8aaf281ad59fbaeae6bcef709" gracePeriod=30
Oct 30 17:12:39.600843 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:39.600801    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409387-8586759869-rvkdt"
Oct 30 17:12:39.667481 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:39.667443    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a59fb466-c9be-42d1-8980-8cf2b28d71be-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a59fb466-c9be-42d1-8980-8cf2b28d71be-maas-api-tls\") pod \"a59fb466-c9be-42d1-8980-8cf2b28d71be\" (UID: \"a59fb466-c9be-42d1-8980-8cf2b28d71be\") "
Oct 30 17:12:39.667759 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:39.667737    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a59fb466-c9be-42d1-8980-8cf2b28d71be-kube-api-access-zhh95\" (UniqueName: \"kubernetes.io/projected/a59fb466-c9be-42d1-8980-8cf2b28d71be-kube-api-access-zhh95\") pod \"a59fb466-c9be-42d1-8980-8cf2b28d71be\" (UID: \"a59fb466-c9be-42d1-8980-8cf2b28d71be\") "
Oct 30 17:12:39.669235 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:39.669212    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a59fb466-c9be-42d1-8980-8cf2b28d71be-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a59fb466-c9be-42d1-8980-8cf2b28d71be-metrics-certs\") pod \"a59fb466-c9be-42d1-8980-8cf2b28d71be\" (UID: \"a59fb466-c9be-42d1-8980-8cf2b28d71be\") "
Oct 30 17:12:39.669971 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:39.669946    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a59fb466-c9be-42d1-8980-8cf2b28d71be-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59fb466-c9be-42d1-8980-8cf2b28d71be-openshift-service-ca\") pod \"a59fb466-c9be-42d1-8980-8cf2b28d71be\" (UID: \"a59fb466-c9be-42d1-8980-8cf2b28d71be\") "
Oct 30 17:12:39.673307 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:39.673283    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a59fb466-c9be-42d1-8980-8cf2b28d71be-openshift-service-ca" pod "a59fb466-c9be-42d1-8980-8cf2b28d71be" (UID: "a59fb466-c9be-42d1-8980-8cf2b28d71be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:12:39.675780 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:39.675737    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a59fb466-c9be-42d1-8980-8cf2b28d71be-kube-api-access-zhh95" pod "a59fb466-c9be-42d1-8980-8cf2b28d71be" (UID: "a59fb466-c9be-42d1-8980-8cf2b28d71be"). InnerVolumeSpecName "kube-api-access-zhh95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:12:39.676281 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:39.676255    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a59fb466-c9be-42d1-8980-8cf2b28d71be-maas-api-tls" pod "a59fb466-c9be-42d1-8980-8cf2b28d71be" (UID: "a59fb466-c9be-42d1-8980-8cf2b28d71be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:12:39.677339 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:39.677315    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a59fb466-c9be-42d1-8980-8cf2b28d71be-metrics-certs" pod "a59fb466-c9be-42d1-8980-8cf2b28d71be" (UID: "a59fb466-c9be-42d1-8980-8cf2b28d71be"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:12:39.775228 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:39.775138    4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59fb466-c9be-42d1-8980-8cf2b28d71be-openshift-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:12:39.775228 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:39.775174    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a59fb466-c9be-42d1-8980-8cf2b28d71be-maas-api-tls\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:12:39.775228 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:39.775189    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhh95\" (UniqueName: \"kubernetes.io/projected/a59fb466-c9be-42d1-8980-8cf2b28d71be-kube-api-access-zhh95\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:12:39.775228 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:39.775202    4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a59fb466-c9be-42d1-8980-8cf2b28d71be-metrics-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:12:39.810154 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:39.810039    4433 generic.go:356] "Generic (PLEG): container finished" podID="a59fb466-c9be-42d1-8980-8cf2b28d71be" containerID="328c5f001fa446f26df5e463300863a0ad20f4e8aaf281ad59fbaeae6bcef709" exitCode=0
Oct 30 17:12:39.810154 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:39.810102    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409387-8586759869-rvkdt"
Oct 30 17:12:39.810494 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:39.810216    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409387-8586759869-rvkdt" event={"ID":"a59fb466-c9be-42d1-8980-8cf2b28d71be","Type":"ContainerDied","Data":"328c5f001fa446f26df5e463300863a0ad20f4e8aaf281ad59fbaeae6bcef709"}
Oct 30 17:12:39.810494 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:39.810268    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-409387-8586759869-rvkdt" event={"ID":"a59fb466-c9be-42d1-8980-8cf2b28d71be","Type":"ContainerDied","Data":"a3387ef071a6dd030ab40c60c97d6100992de6bc566362c8bbee54ea5f1182cd"}
Oct 30 17:12:39.810494 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:39.810295    4433 scope.go:122] "RemoveContainer" containerID="328c5f001fa446f26df5e463300863a0ad20f4e8aaf281ad59fbaeae6bcef709"
Oct 30 17:12:39.864893 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:39.856622    4433 scope.go:122] "RemoveContainer" containerID="328c5f001fa446f26df5e463300863a0ad20f4e8aaf281ad59fbaeae6bcef709"
Oct 30 17:12:39.864893 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:12:39.857121    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"328c5f001fa446f26df5e463300863a0ad20f4e8aaf281ad59fbaeae6bcef709\": container with ID starting with 328c5f001fa446f26df5e463300863a0ad20f4e8aaf281ad59fbaeae6bcef709 not found: ID does not exist" containerID="328c5f001fa446f26df5e463300863a0ad20f4e8aaf281ad59fbaeae6bcef709"
Oct 30 17:12:39.864893 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:39.857165    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"328c5f001fa446f26df5e463300863a0ad20f4e8aaf281ad59fbaeae6bcef709"} err="failed to get container status \"328c5f001fa446f26df5e463300863a0ad20f4e8aaf281ad59fbaeae6bcef709\": rpc error: code = NotFound desc = could not find container \"328c5f001fa446f26df5e463300863a0ad20f4e8aaf281ad59fbaeae6bcef709\": container with ID starting with 328c5f001fa446f26df5e463300863a0ad20f4e8aaf281ad59fbaeae6bcef709 not found: ID does not exist"
Oct 30 17:12:39.888069 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:39.885294    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-409387-8586759869-rvkdt"]
Oct 30 17:12:39.932572 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:39.932523    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-409387-8586759869-rvkdt"]
Oct 30 17:12:41.697358 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:41.697320    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a59fb466-c9be-42d1-8980-8cf2b28d71be" path="/var/lib/kubelet/pods/a59fb466-c9be-42d1-8980-8cf2b28d71be/volumes"
Oct 30 17:12:43.110172 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:43.110123    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-gvpm6" podUID="5b1905ac-5888-4977-b8df-92959b8c4d67" containerName="payload-pre-processing" containerID="cri-o://5c3acc58b0f244707e7b43ea33a809528b82012f29753471eefd5183312c3805" gracePeriod=25
Oct 30 17:12:43.526444 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:43.526390    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-gvpm6"
Oct 30 17:12:43.647088 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:43.646991    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b1905ac-5888-4977-b8df-92959b8c4d67-kube-api-access-g2mbv\" (UniqueName: \"kubernetes.io/projected/5b1905ac-5888-4977-b8df-92959b8c4d67-kube-api-access-g2mbv\") pod \"5b1905ac-5888-4977-b8df-92959b8c4d67\" (UID: \"5b1905ac-5888-4977-b8df-92959b8c4d67\") "
Oct 30 17:12:43.647263 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:43.647152    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b1905ac-5888-4977-b8df-92959b8c4d67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b1905ac-5888-4977-b8df-92959b8c4d67-plugins-config-volume\") pod \"5b1905ac-5888-4977-b8df-92959b8c4d67\" (UID: \"5b1905ac-5888-4977-b8df-92959b8c4d67\") "
Oct 30 17:12:43.647773 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:43.647747    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b1905ac-5888-4977-b8df-92959b8c4d67-plugins-config-volume" pod "5b1905ac-5888-4977-b8df-92959b8c4d67" (UID: "5b1905ac-5888-4977-b8df-92959b8c4d67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:12:43.649990 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:43.649962    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b1905ac-5888-4977-b8df-92959b8c4d67-kube-api-access-g2mbv" pod "5b1905ac-5888-4977-b8df-92959b8c4d67" (UID: "5b1905ac-5888-4977-b8df-92959b8c4d67"). InnerVolumeSpecName "kube-api-access-g2mbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:12:43.749979 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:43.749946    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2mbv\" (UniqueName: \"kubernetes.io/projected/5b1905ac-5888-4977-b8df-92959b8c4d67-kube-api-access-g2mbv\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:12:43.749979 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:43.749974    4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b1905ac-5888-4977-b8df-92959b8c4d67-plugins-config-volume\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:12:43.839683 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:43.839649    4433 generic.go:356] "Generic (PLEG): container finished" podID="5b1905ac-5888-4977-b8df-92959b8c4d67" containerID="5c3acc58b0f244707e7b43ea33a809528b82012f29753471eefd5183312c3805" exitCode=0
Oct 30 17:12:43.839868 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:43.839699    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-gvpm6" event={"ID":"5b1905ac-5888-4977-b8df-92959b8c4d67","Type":"ContainerDied","Data":"5c3acc58b0f244707e7b43ea33a809528b82012f29753471eefd5183312c3805"}
Oct 30 17:12:43.839868 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:43.839747    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-gvpm6"
Oct 30 17:12:43.839868 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:43.839752    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-gvpm6" event={"ID":"5b1905ac-5888-4977-b8df-92959b8c4d67","Type":"ContainerDied","Data":"3a9bd3bb347fcc30684074143a948bf3ab4a9cd7e01c36add2554165146f4d0c"}
Oct 30 17:12:43.839868 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:43.839781    4433 scope.go:122] "RemoveContainer" containerID="5c3acc58b0f244707e7b43ea33a809528b82012f29753471eefd5183312c3805"
Oct 30 17:12:43.850918 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:43.850891    4433 scope.go:122] "RemoveContainer" containerID="5c3acc58b0f244707e7b43ea33a809528b82012f29753471eefd5183312c3805"
Oct 30 17:12:43.851228 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:12:43.851207    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c3acc58b0f244707e7b43ea33a809528b82012f29753471eefd5183312c3805\": container with ID starting with 5c3acc58b0f244707e7b43ea33a809528b82012f29753471eefd5183312c3805 not found: ID does not exist" containerID="5c3acc58b0f244707e7b43ea33a809528b82012f29753471eefd5183312c3805"
Oct 30 17:12:43.851313 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:43.851235    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c3acc58b0f244707e7b43ea33a809528b82012f29753471eefd5183312c3805"} err="failed to get container status \"5c3acc58b0f244707e7b43ea33a809528b82012f29753471eefd5183312c3805\": rpc error: code = NotFound desc = could not find container \"5c3acc58b0f244707e7b43ea33a809528b82012f29753471eefd5183312c3805\": container with ID starting with 5c3acc58b0f244707e7b43ea33a809528b82012f29753471eefd5183312c3805 not found: ID does not exist"
Oct 30 17:12:43.891514 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:43.891479    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-gvpm6"]
Oct 30 17:12:43.902514 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:43.902444    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-gvpm6"]
Oct 30 17:12:44.417694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:44.417646    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-gvpm6" podUID="5b1905ac-5888-4977-b8df-92959b8c4d67" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.104:50052\" within 1s: context deadline exceeded"
Oct 30 17:12:44.418139 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:12:44.417780    4433 logging.go:55] [core] [Channel #170 SubChannel #171]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.104:50052", ServerName: "10.133.0.104:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.104:50052: operation was canceled"
Oct 30 17:12:45.698422 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:45.698370    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b1905ac-5888-4977-b8df-92959b8c4d67" path="/var/lib/kubelet/pods/5b1905ac-5888-4977-b8df-92959b8c4d67/volumes"
Oct 30 17:12:48.882038 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:48.881998    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"]
Oct 30 17:12:48.882624 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:48.882243    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp" podUID="83c7d819-6442-4e39-b1cc-bb36c62d5732" containerName="istio-proxy" containerID="cri-o://5159b83fa70a8a76b0935192adfcdbe5f84563c87e1b567b4cf980aa3b822f81" gracePeriod=30
Oct 30 17:12:48.975173 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:48.975137    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:12:54.189001 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.188975    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:12:54.317816 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.317770    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83c7d819-6442-4e39-b1cc-bb36c62d5732-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c7d819-6442-4e39-b1cc-bb36c62d5732-istio-podinfo\") pod \"83c7d819-6442-4e39-b1cc-bb36c62d5732\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") "
Oct 30 17:12:54.318161 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.318131    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83c7d819-6442-4e39-b1cc-bb36c62d5732-kube-api-access-zm2hj\" (UniqueName: \"kubernetes.io/projected/83c7d819-6442-4e39-b1cc-bb36c62d5732-kube-api-access-zm2hj\") pod \"83c7d819-6442-4e39-b1cc-bb36c62d5732\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") "
Oct 30 17:12:54.319387 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.318255    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c7d819-6442-4e39-b1cc-bb36c62d5732-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c7d819-6442-4e39-b1cc-bb36c62d5732-credential-socket\") pod \"83c7d819-6442-4e39-b1cc-bb36c62d5732\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") "
Oct 30 17:12:54.319387 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.318384    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c7d819-6442-4e39-b1cc-bb36c62d5732-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c7d819-6442-4e39-b1cc-bb36c62d5732-workload-socket\") pod \"83c7d819-6442-4e39-b1cc-bb36c62d5732\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") "
Oct 30 17:12:54.319387 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.318463    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83c7d819-6442-4e39-b1cc-bb36c62d5732-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c7d819-6442-4e39-b1cc-bb36c62d5732-istiod-ca-cert\") pod \"83c7d819-6442-4e39-b1cc-bb36c62d5732\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") "
Oct 30 17:12:54.319387 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.318495    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c7d819-6442-4e39-b1cc-bb36c62d5732-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c7d819-6442-4e39-b1cc-bb36c62d5732-workload-certs\") pod \"83c7d819-6442-4e39-b1cc-bb36c62d5732\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") "
Oct 30 17:12:54.319387 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.318547    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83c7d819-6442-4e39-b1cc-bb36c62d5732-istio-token\" (UniqueName: \"kubernetes.io/projected/83c7d819-6442-4e39-b1cc-bb36c62d5732-istio-token\") pod \"83c7d819-6442-4e39-b1cc-bb36c62d5732\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") "
Oct 30 17:12:54.319387 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.318612    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c7d819-6442-4e39-b1cc-bb36c62d5732-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c7d819-6442-4e39-b1cc-bb36c62d5732-istio-envoy\") pod \"83c7d819-6442-4e39-b1cc-bb36c62d5732\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") "
Oct 30 17:12:54.319387 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.318713    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c7d819-6442-4e39-b1cc-bb36c62d5732-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c7d819-6442-4e39-b1cc-bb36c62d5732-istio-data\") pod \"83c7d819-6442-4e39-b1cc-bb36c62d5732\" (UID: \"83c7d819-6442-4e39-b1cc-bb36c62d5732\") "
Oct 30 17:12:54.320216 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.319390    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c7d819-6442-4e39-b1cc-bb36c62d5732-istiod-ca-cert" pod "83c7d819-6442-4e39-b1cc-bb36c62d5732" (UID: "83c7d819-6442-4e39-b1cc-bb36c62d5732"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:12:54.320216 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.319661    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c7d819-6442-4e39-b1cc-bb36c62d5732-workload-socket" pod "83c7d819-6442-4e39-b1cc-bb36c62d5732" (UID: "83c7d819-6442-4e39-b1cc-bb36c62d5732"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:12:54.320216 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.319697    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c7d819-6442-4e39-b1cc-bb36c62d5732-workload-certs" pod "83c7d819-6442-4e39-b1cc-bb36c62d5732" (UID: "83c7d819-6442-4e39-b1cc-bb36c62d5732"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:12:54.320216 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.319728    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c7d819-6442-4e39-b1cc-bb36c62d5732-credential-socket" pod "83c7d819-6442-4e39-b1cc-bb36c62d5732" (UID: "83c7d819-6442-4e39-b1cc-bb36c62d5732"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:12:54.320216 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.319801    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c7d819-6442-4e39-b1cc-bb36c62d5732-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:12:54.320216 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.320077    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c7d819-6442-4e39-b1cc-bb36c62d5732-istio-data" pod "83c7d819-6442-4e39-b1cc-bb36c62d5732" (UID: "83c7d819-6442-4e39-b1cc-bb36c62d5732"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:12:54.322666 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.322624    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83c7d819-6442-4e39-b1cc-bb36c62d5732-istio-podinfo" pod "83c7d819-6442-4e39-b1cc-bb36c62d5732" (UID: "83c7d819-6442-4e39-b1cc-bb36c62d5732"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:12:54.337833 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.323254    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c7d819-6442-4e39-b1cc-bb36c62d5732-kube-api-access-zm2hj" pod "83c7d819-6442-4e39-b1cc-bb36c62d5732" (UID: "83c7d819-6442-4e39-b1cc-bb36c62d5732"). InnerVolumeSpecName "kube-api-access-zm2hj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:12:54.349314 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.348496    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c7d819-6442-4e39-b1cc-bb36c62d5732-istio-token" pod "83c7d819-6442-4e39-b1cc-bb36c62d5732" (UID: "83c7d819-6442-4e39-b1cc-bb36c62d5732"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:12:54.349314 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.348598    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c7d819-6442-4e39-b1cc-bb36c62d5732-istio-envoy" pod "83c7d819-6442-4e39-b1cc-bb36c62d5732" (UID: "83c7d819-6442-4e39-b1cc-bb36c62d5732"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:12:54.421796 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.421697    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c7d819-6442-4e39-b1cc-bb36c62d5732-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:12:54.421796 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.421729    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c7d819-6442-4e39-b1cc-bb36c62d5732-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:12:54.421796 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.421739    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c7d819-6442-4e39-b1cc-bb36c62d5732-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:12:54.421796 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.421746    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c7d819-6442-4e39-b1cc-bb36c62d5732-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:12:54.421796 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.421754    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c7d819-6442-4e39-b1cc-bb36c62d5732-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:12:54.421796 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.421762    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c7d819-6442-4e39-b1cc-bb36c62d5732-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:12:54.421796 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.421771    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm2hj\" (UniqueName: \"kubernetes.io/projected/83c7d819-6442-4e39-b1cc-bb36c62d5732-kube-api-access-zm2hj\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:12:54.421796 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.421780    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c7d819-6442-4e39-b1cc-bb36c62d5732-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:12:54.933227 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.933188    4433 generic.go:356] "Generic (PLEG): container finished" podID="83c7d819-6442-4e39-b1cc-bb36c62d5732" containerID="5159b83fa70a8a76b0935192adfcdbe5f84563c87e1b567b4cf980aa3b822f81" exitCode=0
Oct 30 17:12:54.933419 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.933254    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"
Oct 30 17:12:54.933419 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.933274    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp" event={"ID":"83c7d819-6442-4e39-b1cc-bb36c62d5732","Type":"ContainerDied","Data":"5159b83fa70a8a76b0935192adfcdbe5f84563c87e1b567b4cf980aa3b822f81"}
Oct 30 17:12:54.933419 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.933337    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp" event={"ID":"83c7d819-6442-4e39-b1cc-bb36c62d5732","Type":"ContainerDied","Data":"62022352f2746aa1eff843a66ee0eb2ada57ec3cdd966990bf672563cfa111d7"}
Oct 30 17:12:54.933419 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.933348    4433 scope.go:122] "RemoveContainer" containerID="5159b83fa70a8a76b0935192adfcdbe5f84563c87e1b567b4cf980aa3b822f81"
Oct 30 17:12:54.945594 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.945568    4433 scope.go:122] "RemoveContainer" containerID="078e67a901fc5950d2f84bcfdc4a6629a0a9cd2d9f1223d78bb1ce68f6fd71db"
Oct 30 17:12:54.957371 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.957353    4433 scope.go:122] "RemoveContainer" containerID="5159b83fa70a8a76b0935192adfcdbe5f84563c87e1b567b4cf980aa3b822f81"
Oct 30 17:12:54.958022 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:12:54.957900    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5159b83fa70a8a76b0935192adfcdbe5f84563c87e1b567b4cf980aa3b822f81\": container with ID starting with 5159b83fa70a8a76b0935192adfcdbe5f84563c87e1b567b4cf980aa3b822f81 not found: ID does not exist" containerID="5159b83fa70a8a76b0935192adfcdbe5f84563c87e1b567b4cf980aa3b822f81"
Oct 30 17:12:54.958022 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.957937    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5159b83fa70a8a76b0935192adfcdbe5f84563c87e1b567b4cf980aa3b822f81"} err="failed to get container status \"5159b83fa70a8a76b0935192adfcdbe5f84563c87e1b567b4cf980aa3b822f81\": rpc error: code = NotFound desc = could not find container \"5159b83fa70a8a76b0935192adfcdbe5f84563c87e1b567b4cf980aa3b822f81\": container with ID starting with 5159b83fa70a8a76b0935192adfcdbe5f84563c87e1b567b4cf980aa3b822f81 not found: ID does not exist"
Oct 30 17:12:54.958022 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.957960    4433 scope.go:122] "RemoveContainer" containerID="078e67a901fc5950d2f84bcfdc4a6629a0a9cd2d9f1223d78bb1ce68f6fd71db"
Oct 30 17:12:54.958372 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:12:54.958303    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"078e67a901fc5950d2f84bcfdc4a6629a0a9cd2d9f1223d78bb1ce68f6fd71db\": container with ID starting with 078e67a901fc5950d2f84bcfdc4a6629a0a9cd2d9f1223d78bb1ce68f6fd71db not found: ID does not exist" containerID="078e67a901fc5950d2f84bcfdc4a6629a0a9cd2d9f1223d78bb1ce68f6fd71db"
Oct 30 17:12:54.958372 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.958346    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"078e67a901fc5950d2f84bcfdc4a6629a0a9cd2d9f1223d78bb1ce68f6fd71db"} err="failed to get container status \"078e67a901fc5950d2f84bcfdc4a6629a0a9cd2d9f1223d78bb1ce68f6fd71db\": rpc error: code = NotFound desc = could not find container \"078e67a901fc5950d2f84bcfdc4a6629a0a9cd2d9f1223d78bb1ce68f6fd71db\": container with ID starting with 078e67a901fc5950d2f84bcfdc4a6629a0a9cd2d9f1223d78bb1ce68f6fd71db not found: ID does not exist"
Oct 30 17:12:54.999633 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:54.999599    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"]
Oct 30 17:12:55.011622 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:55.011590    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-409387-openshift-default-754d8fcc59-sdmlp"]
Oct 30 17:12:55.697256 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:55.697220    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83c7d819-6442-4e39-b1cc-bb36c62d5732" path="/var/lib/kubelet/pods/83c7d819-6442-4e39-b1cc-bb36c62d5732/volumes"
Oct 30 17:12:56.979285 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:12:56.979248    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:12:59.565002 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:12:59.564964    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c4c2e4-7594c7fb59-nnq5t"]
Oct 30 17:13:04.565311 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:04.565268    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c2e4-7594c7fb59-nnq5t" podUID="56a5d068-44be-4afa-8811-b9b89f5dd25e" containerName="payload-pre-processing" containerID="cri-o://cf88b2b763b3a5e2876c9e16719e87c85a231c40e9f1eb5101cef3c205530370" gracePeriod=25
Oct 30 17:13:04.889801 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:04.889773    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c2e4-7594c7fb59-nnq5t"
Oct 30 17:13:04.909269 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:04.909224    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56a5d068-44be-4afa-8811-b9b89f5dd25e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56a5d068-44be-4afa-8811-b9b89f5dd25e-plugins-config-volume\") pod \"56a5d068-44be-4afa-8811-b9b89f5dd25e\" (UID: \"56a5d068-44be-4afa-8811-b9b89f5dd25e\") "
Oct 30 17:13:04.909558 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:04.909364    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56a5d068-44be-4afa-8811-b9b89f5dd25e-kube-api-access-lnzk2\" (UniqueName: \"kubernetes.io/projected/56a5d068-44be-4afa-8811-b9b89f5dd25e-kube-api-access-lnzk2\") pod \"56a5d068-44be-4afa-8811-b9b89f5dd25e\" (UID: \"56a5d068-44be-4afa-8811-b9b89f5dd25e\") "
Oct 30 17:13:04.909820 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:04.909791    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56a5d068-44be-4afa-8811-b9b89f5dd25e-plugins-config-volume" pod "56a5d068-44be-4afa-8811-b9b89f5dd25e" (UID: "56a5d068-44be-4afa-8811-b9b89f5dd25e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:13:04.910628 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:04.910604    4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56a5d068-44be-4afa-8811-b9b89f5dd25e-plugins-config-volume\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:04.912896 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:04.912871    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a5d068-44be-4afa-8811-b9b89f5dd25e-kube-api-access-lnzk2" pod "56a5d068-44be-4afa-8811-b9b89f5dd25e" (UID: "56a5d068-44be-4afa-8811-b9b89f5dd25e"). InnerVolumeSpecName "kube-api-access-lnzk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:13:05.006426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:05.006364    4433 generic.go:356] "Generic (PLEG): container finished" podID="56a5d068-44be-4afa-8811-b9b89f5dd25e" containerID="cf88b2b763b3a5e2876c9e16719e87c85a231c40e9f1eb5101cef3c205530370" exitCode=0
Oct 30 17:13:05.006597 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:05.006454    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c2e4-7594c7fb59-nnq5t" event={"ID":"56a5d068-44be-4afa-8811-b9b89f5dd25e","Type":"ContainerDied","Data":"cf88b2b763b3a5e2876c9e16719e87c85a231c40e9f1eb5101cef3c205530370"}
Oct 30 17:13:05.006597 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:05.006509    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c2e4-7594c7fb59-nnq5t" event={"ID":"56a5d068-44be-4afa-8811-b9b89f5dd25e","Type":"ContainerDied","Data":"cc2da9b18280fc0f603a1710f91cb9880af98d7e2a9b0e2d34774a896d83865e"}
Oct 30 17:13:05.006597 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:05.006529    4433 scope.go:122] "RemoveContainer" containerID="cf88b2b763b3a5e2876c9e16719e87c85a231c40e9f1eb5101cef3c205530370"
Oct 30 17:13:05.006791 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:05.006512    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c2e4-7594c7fb59-nnq5t"
Oct 30 17:13:05.011889 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:05.011864    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnzk2\" (UniqueName: \"kubernetes.io/projected/56a5d068-44be-4afa-8811-b9b89f5dd25e-kube-api-access-lnzk2\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:05.019167 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:05.019148    4433 scope.go:122] "RemoveContainer" containerID="cf88b2b763b3a5e2876c9e16719e87c85a231c40e9f1eb5101cef3c205530370"
Oct 30 17:13:05.019530 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:13:05.019508    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf88b2b763b3a5e2876c9e16719e87c85a231c40e9f1eb5101cef3c205530370\": container with ID starting with cf88b2b763b3a5e2876c9e16719e87c85a231c40e9f1eb5101cef3c205530370 not found: ID does not exist" containerID="cf88b2b763b3a5e2876c9e16719e87c85a231c40e9f1eb5101cef3c205530370"
Oct 30 17:13:05.019602 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:05.019540    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf88b2b763b3a5e2876c9e16719e87c85a231c40e9f1eb5101cef3c205530370"} err="failed to get container status \"cf88b2b763b3a5e2876c9e16719e87c85a231c40e9f1eb5101cef3c205530370\": rpc error: code = NotFound desc = could not find container \"cf88b2b763b3a5e2876c9e16719e87c85a231c40e9f1eb5101cef3c205530370\": container with ID starting with cf88b2b763b3a5e2876c9e16719e87c85a231c40e9f1eb5101cef3c205530370 not found: ID does not exist"
Oct 30 17:13:05.059550 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:05.059517    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c4c2e4-7594c7fb59-nnq5t"]
Oct 30 17:13:05.064791 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:05.064763    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c4c2e4-7594c7fb59-nnq5t"]
Oct 30 17:13:05.697227 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:05.697195    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56a5d068-44be-4afa-8811-b9b89f5dd25e" path="/var/lib/kubelet/pods/56a5d068-44be-4afa-8811-b9b89f5dd25e/volumes"
Oct 30 17:13:05.874969 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:05.874934    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"]
Oct 30 17:13:09.134119 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:09.134081    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"]
Oct 30 17:13:09.134658 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:09.134315    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz" podUID="e6431be3-b51d-40bc-bc87-ddf189905e14" containerName="istio-proxy" containerID="cri-o://f15641e0650eb18d0e9ac76f8e575f111731df480cebe92de1b855a61cab3a82" gracePeriod=30
Oct 30 17:13:09.385237 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:09.385150    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"]
Oct 30 17:13:09.385470 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:09.385419    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm" podUID="5b04a22a-0651-49fe-9d44-7c0067c2ad14" containerName="istio-proxy" containerID="cri-o://b462e2478aa9042db5e06f2b1a383ea7f4f79460671163bb4104fdd809203b85" gracePeriod=30
Oct 30 17:13:09.680294 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:09.680260    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"]
Oct 30 17:13:09.680553 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:09.680479    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw" podUID="1a82e355-1770-48e7-a8ab-92d5d1b6d257" containerName="istio-proxy" containerID="cri-o://adc5cfb160ede6a8214cee4f1864243b93292e6761062e1809b31e6367de7ed2" gracePeriod=30
Oct 30 17:13:09.803688 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:09.803648    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"]
Oct 30 17:13:09.803996 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:09.803960    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479" podUID="20dd24a1-9e9c-40d2-8ad8-bf0a206f292e" containerName="istio-proxy" containerID="cri-o://50f27108e1e0a0f9bf20bcfd12c55313d37647207fa3d6d2c4714cac324afed8" gracePeriod=30
Oct 30 17:13:14.420148 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.420077    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"
Oct 30 17:13:14.474169 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.474136    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6431be3-b51d-40bc-bc87-ddf189905e14-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6431be3-b51d-40bc-bc87-ddf189905e14-istiod-ca-cert\") pod \"e6431be3-b51d-40bc-bc87-ddf189905e14\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") "
Oct 30 17:13:14.474334 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.474229    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6431be3-b51d-40bc-bc87-ddf189905e14-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6431be3-b51d-40bc-bc87-ddf189905e14-istio-podinfo\") pod \"e6431be3-b51d-40bc-bc87-ddf189905e14\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") "
Oct 30 17:13:14.474334 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.474331    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6431be3-b51d-40bc-bc87-ddf189905e14-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6431be3-b51d-40bc-bc87-ddf189905e14-credential-socket\") pod \"e6431be3-b51d-40bc-bc87-ddf189905e14\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") "
Oct 30 17:13:14.474512 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.474353    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6431be3-b51d-40bc-bc87-ddf189905e14-kube-api-access-fvcc7\" (UniqueName: \"kubernetes.io/projected/e6431be3-b51d-40bc-bc87-ddf189905e14-kube-api-access-fvcc7\") pod \"e6431be3-b51d-40bc-bc87-ddf189905e14\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") "
Oct 30 17:13:14.474512 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.474381    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6431be3-b51d-40bc-bc87-ddf189905e14-istio-token\" (UniqueName: \"kubernetes.io/projected/e6431be3-b51d-40bc-bc87-ddf189905e14-istio-token\") pod \"e6431be3-b51d-40bc-bc87-ddf189905e14\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") "
Oct 30 17:13:14.474512 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.474442    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6431be3-b51d-40bc-bc87-ddf189905e14-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6431be3-b51d-40bc-bc87-ddf189905e14-workload-certs\") pod \"e6431be3-b51d-40bc-bc87-ddf189905e14\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") "
Oct 30 17:13:14.474664 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.474520    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6431be3-b51d-40bc-bc87-ddf189905e14-istiod-ca-cert" pod "e6431be3-b51d-40bc-bc87-ddf189905e14" (UID: "e6431be3-b51d-40bc-bc87-ddf189905e14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:13:14.474664 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.474529    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6431be3-b51d-40bc-bc87-ddf189905e14-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6431be3-b51d-40bc-bc87-ddf189905e14-workload-socket\") pod \"e6431be3-b51d-40bc-bc87-ddf189905e14\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") "
Oct 30 17:13:14.474825 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.474658    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6431be3-b51d-40bc-bc87-ddf189905e14-credential-socket" pod "e6431be3-b51d-40bc-bc87-ddf189905e14" (UID: "e6431be3-b51d-40bc-bc87-ddf189905e14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:13:14.475319 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.474732    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6431be3-b51d-40bc-bc87-ddf189905e14-workload-socket" pod "e6431be3-b51d-40bc-bc87-ddf189905e14" (UID: "e6431be3-b51d-40bc-bc87-ddf189905e14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:13:14.476059 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.474773    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6431be3-b51d-40bc-bc87-ddf189905e14-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6431be3-b51d-40bc-bc87-ddf189905e14-istio-data\") pod \"e6431be3-b51d-40bc-bc87-ddf189905e14\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") "
Oct 30 17:13:14.476059 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.475443    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6431be3-b51d-40bc-bc87-ddf189905e14-istio-data" pod "e6431be3-b51d-40bc-bc87-ddf189905e14" (UID: "e6431be3-b51d-40bc-bc87-ddf189905e14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:13:14.476059 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.475106    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6431be3-b51d-40bc-bc87-ddf189905e14-workload-certs" pod "e6431be3-b51d-40bc-bc87-ddf189905e14" (UID: "e6431be3-b51d-40bc-bc87-ddf189905e14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:13:14.476059 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.475486    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6431be3-b51d-40bc-bc87-ddf189905e14-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6431be3-b51d-40bc-bc87-ddf189905e14-istio-envoy\") pod \"e6431be3-b51d-40bc-bc87-ddf189905e14\" (UID: \"e6431be3-b51d-40bc-bc87-ddf189905e14\") "
Oct 30 17:13:14.477671 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.477539    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6431be3-b51d-40bc-bc87-ddf189905e14-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:14.477671 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.477567    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6431be3-b51d-40bc-bc87-ddf189905e14-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:14.477671 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.477582    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6431be3-b51d-40bc-bc87-ddf189905e14-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:14.477671 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.477595    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6431be3-b51d-40bc-bc87-ddf189905e14-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:14.477671 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.477608    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6431be3-b51d-40bc-bc87-ddf189905e14-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:14.477671 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.477630    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6431be3-b51d-40bc-bc87-ddf189905e14-istio-podinfo" pod "e6431be3-b51d-40bc-bc87-ddf189905e14" (UID: "e6431be3-b51d-40bc-bc87-ddf189905e14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:13:14.477943 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.477685    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6431be3-b51d-40bc-bc87-ddf189905e14-kube-api-access-fvcc7" pod "e6431be3-b51d-40bc-bc87-ddf189905e14" (UID: "e6431be3-b51d-40bc-bc87-ddf189905e14"). InnerVolumeSpecName "kube-api-access-fvcc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:13:14.477943 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.477717    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6431be3-b51d-40bc-bc87-ddf189905e14-istio-token" pod "e6431be3-b51d-40bc-bc87-ddf189905e14" (UID: "e6431be3-b51d-40bc-bc87-ddf189905e14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:13:14.478335 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.478302    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6431be3-b51d-40bc-bc87-ddf189905e14-istio-envoy" pod "e6431be3-b51d-40bc-bc87-ddf189905e14" (UID: "e6431be3-b51d-40bc-bc87-ddf189905e14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:13:14.579315 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.579275    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6431be3-b51d-40bc-bc87-ddf189905e14-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:14.579315 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.579305    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvcc7\" (UniqueName: \"kubernetes.io/projected/e6431be3-b51d-40bc-bc87-ddf189905e14-kube-api-access-fvcc7\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:14.579315 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.579316    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6431be3-b51d-40bc-bc87-ddf189905e14-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:14.579315 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.579325    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6431be3-b51d-40bc-bc87-ddf189905e14-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:14.696457 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.696377    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"
Oct 30 17:13:14.785199 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.784979    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b04a22a-0651-49fe-9d44-7c0067c2ad14-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b04a22a-0651-49fe-9d44-7c0067c2ad14-istio-data\") pod \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") "
Oct 30 17:13:14.785199 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.785052    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b04a22a-0651-49fe-9d44-7c0067c2ad14-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b04a22a-0651-49fe-9d44-7c0067c2ad14-credential-socket\") pod \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") "
Oct 30 17:13:14.785199 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.785116    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b04a22a-0651-49fe-9d44-7c0067c2ad14-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b04a22a-0651-49fe-9d44-7c0067c2ad14-workload-socket\") pod \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") "
Oct 30 17:13:14.785600 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.785259    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b04a22a-0651-49fe-9d44-7c0067c2ad14-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b04a22a-0651-49fe-9d44-7c0067c2ad14-istio-podinfo\") pod \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") "
Oct 30 17:13:14.785600 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.785303    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b04a22a-0651-49fe-9d44-7c0067c2ad14-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b04a22a-0651-49fe-9d44-7c0067c2ad14-istiod-ca-cert\") pod \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") "
Oct 30 17:13:14.785600 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.785339    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b04a22a-0651-49fe-9d44-7c0067c2ad14-istio-token\" (UniqueName: \"kubernetes.io/projected/5b04a22a-0651-49fe-9d44-7c0067c2ad14-istio-token\") pod \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") "
Oct 30 17:13:14.785600 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.785392    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b04a22a-0651-49fe-9d44-7c0067c2ad14-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b04a22a-0651-49fe-9d44-7c0067c2ad14-workload-certs\") pod \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") "
Oct 30 17:13:14.785600 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.785570    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b04a22a-0651-49fe-9d44-7c0067c2ad14-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b04a22a-0651-49fe-9d44-7c0067c2ad14-istio-envoy\") pod \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") "
Oct 30 17:13:14.785855 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.785610    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b04a22a-0651-49fe-9d44-7c0067c2ad14-kube-api-access-nknwp\" (UniqueName: \"kubernetes.io/projected/5b04a22a-0651-49fe-9d44-7c0067c2ad14-kube-api-access-nknwp\") pod \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\" (UID: \"5b04a22a-0651-49fe-9d44-7c0067c2ad14\") "
Oct 30 17:13:14.787200 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.787174    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b04a22a-0651-49fe-9d44-7c0067c2ad14-credential-socket" pod "5b04a22a-0651-49fe-9d44-7c0067c2ad14" (UID: "5b04a22a-0651-49fe-9d44-7c0067c2ad14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:13:14.787548 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.787398    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b04a22a-0651-49fe-9d44-7c0067c2ad14-istio-data" pod "5b04a22a-0651-49fe-9d44-7c0067c2ad14" (UID: "5b04a22a-0651-49fe-9d44-7c0067c2ad14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:13:14.788339 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.788305    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b04a22a-0651-49fe-9d44-7c0067c2ad14-workload-certs" pod "5b04a22a-0651-49fe-9d44-7c0067c2ad14" (UID: "5b04a22a-0651-49fe-9d44-7c0067c2ad14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:13:14.788488 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.788372    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b04a22a-0651-49fe-9d44-7c0067c2ad14-workload-socket" pod "5b04a22a-0651-49fe-9d44-7c0067c2ad14" (UID: "5b04a22a-0651-49fe-9d44-7c0067c2ad14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:13:14.788488 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.788458    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b04a22a-0651-49fe-9d44-7c0067c2ad14-istio-podinfo" pod "5b04a22a-0651-49fe-9d44-7c0067c2ad14" (UID: "5b04a22a-0651-49fe-9d44-7c0067c2ad14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:13:14.788929 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.788909    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b04a22a-0651-49fe-9d44-7c0067c2ad14-istiod-ca-cert" pod "5b04a22a-0651-49fe-9d44-7c0067c2ad14" (UID: "5b04a22a-0651-49fe-9d44-7c0067c2ad14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:13:14.789076 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.789049    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b04a22a-0651-49fe-9d44-7c0067c2ad14-kube-api-access-nknwp" pod "5b04a22a-0651-49fe-9d44-7c0067c2ad14" (UID: "5b04a22a-0651-49fe-9d44-7c0067c2ad14"). InnerVolumeSpecName "kube-api-access-nknwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:13:14.789838 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.789816    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b04a22a-0651-49fe-9d44-7c0067c2ad14-istio-token" pod "5b04a22a-0651-49fe-9d44-7c0067c2ad14" (UID: "5b04a22a-0651-49fe-9d44-7c0067c2ad14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:13:14.790081 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.790056    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b04a22a-0651-49fe-9d44-7c0067c2ad14-istio-envoy" pod "5b04a22a-0651-49fe-9d44-7c0067c2ad14" (UID: "5b04a22a-0651-49fe-9d44-7c0067c2ad14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:13:14.892632 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.892599    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b04a22a-0651-49fe-9d44-7c0067c2ad14-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:14.892632 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.892630    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b04a22a-0651-49fe-9d44-7c0067c2ad14-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:14.892856 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.892642    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b04a22a-0651-49fe-9d44-7c0067c2ad14-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:14.892856 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.892665    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b04a22a-0651-49fe-9d44-7c0067c2ad14-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:14.892856 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.892682    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b04a22a-0651-49fe-9d44-7c0067c2ad14-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:14.892856 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.892690    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b04a22a-0651-49fe-9d44-7c0067c2ad14-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:14.892856 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.892698    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b04a22a-0651-49fe-9d44-7c0067c2ad14-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:14.892856 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.892708    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b04a22a-0651-49fe-9d44-7c0067c2ad14-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:14.892856 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.892719    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nknwp\" (UniqueName: \"kubernetes.io/projected/5b04a22a-0651-49fe-9d44-7c0067c2ad14-kube-api-access-nknwp\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:14.977152 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:14.977122    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"
Oct 30 17:13:15.081045 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.081000    4433 generic.go:356] "Generic (PLEG): container finished" podID="5b04a22a-0651-49fe-9d44-7c0067c2ad14" containerID="b462e2478aa9042db5e06f2b1a383ea7f4f79460671163bb4104fdd809203b85" exitCode=0
Oct 30 17:13:15.081236 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.081135    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm" event={"ID":"5b04a22a-0651-49fe-9d44-7c0067c2ad14","Type":"ContainerDied","Data":"b462e2478aa9042db5e06f2b1a383ea7f4f79460671163bb4104fdd809203b85"}
Oct 30 17:13:15.081236 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.081222    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm" event={"ID":"5b04a22a-0651-49fe-9d44-7c0067c2ad14","Type":"ContainerDied","Data":"2cac68823eeede402ae71594a60d3e32e8173c667f07a8770393a122b57f01b0"}
Oct 30 17:13:15.081369 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.081253    4433 scope.go:122] "RemoveContainer" containerID="b462e2478aa9042db5e06f2b1a383ea7f4f79460671163bb4104fdd809203b85"
Oct 30 17:13:15.081475 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.081457    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"
Oct 30 17:13:15.087117 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.086717    4433 generic.go:356] "Generic (PLEG): container finished" podID="1a82e355-1770-48e7-a8ab-92d5d1b6d257" containerID="adc5cfb160ede6a8214cee4f1864243b93292e6761062e1809b31e6367de7ed2" exitCode=0
Oct 30 17:13:15.087117 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.086822    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw" event={"ID":"1a82e355-1770-48e7-a8ab-92d5d1b6d257","Type":"ContainerDied","Data":"adc5cfb160ede6a8214cee4f1864243b93292e6761062e1809b31e6367de7ed2"}
Oct 30 17:13:15.087117 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.086868    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw" event={"ID":"1a82e355-1770-48e7-a8ab-92d5d1b6d257","Type":"ContainerDied","Data":"cdcb8d9e44eab83b54ecca8a28d3071de80a63fc8f09f4a4f66bf04f928dde08"}
Oct 30 17:13:15.087117 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.086950    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"
Oct 30 17:13:15.090683 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.090651    4433 generic.go:356] "Generic (PLEG): container finished" podID="e6431be3-b51d-40bc-bc87-ddf189905e14" containerID="f15641e0650eb18d0e9ac76f8e575f111731df480cebe92de1b855a61cab3a82" exitCode=0
Oct 30 17:13:15.090802 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.090779    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz" event={"ID":"e6431be3-b51d-40bc-bc87-ddf189905e14","Type":"ContainerDied","Data":"f15641e0650eb18d0e9ac76f8e575f111731df480cebe92de1b855a61cab3a82"}
Oct 30 17:13:15.090853 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.090841    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"
Oct 30 17:13:15.090932 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.090841    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz" event={"ID":"e6431be3-b51d-40bc-bc87-ddf189905e14","Type":"ContainerDied","Data":"e3c513ec4ce23db0c68627aa3b4f298c145c083e01887ba76ec9f436aa7372a2"}
Oct 30 17:13:15.092647 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.092620    4433 generic.go:356] "Generic (PLEG): container finished" podID="20dd24a1-9e9c-40d2-8ad8-bf0a206f292e" containerID="50f27108e1e0a0f9bf20bcfd12c55313d37647207fa3d6d2c4714cac324afed8" exitCode=0
Oct 30 17:13:15.092753 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.092724    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479" event={"ID":"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e","Type":"ContainerDied","Data":"50f27108e1e0a0f9bf20bcfd12c55313d37647207fa3d6d2c4714cac324afed8"}
Oct 30 17:13:15.095248 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.095226    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a82e355-1770-48e7-a8ab-92d5d1b6d257-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a82e355-1770-48e7-a8ab-92d5d1b6d257-istio-envoy\") pod \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") "
Oct 30 17:13:15.095364 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.095312    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a82e355-1770-48e7-a8ab-92d5d1b6d257-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a82e355-1770-48e7-a8ab-92d5d1b6d257-istiod-ca-cert\") pod \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") "
Oct 30 17:13:15.095498 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.095387    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a82e355-1770-48e7-a8ab-92d5d1b6d257-istio-token\" (UniqueName: \"kubernetes.io/projected/1a82e355-1770-48e7-a8ab-92d5d1b6d257-istio-token\") pod \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") "
Oct 30 17:13:15.095498 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.095442    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a82e355-1770-48e7-a8ab-92d5d1b6d257-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a82e355-1770-48e7-a8ab-92d5d1b6d257-workload-socket\") pod \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") "
Oct 30 17:13:15.095498 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.095485    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a82e355-1770-48e7-a8ab-92d5d1b6d257-kube-api-access-tj6rj\" (UniqueName: \"kubernetes.io/projected/1a82e355-1770-48e7-a8ab-92d5d1b6d257-kube-api-access-tj6rj\") pod \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") "
Oct 30 17:13:15.095646 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.095528    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a82e355-1770-48e7-a8ab-92d5d1b6d257-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a82e355-1770-48e7-a8ab-92d5d1b6d257-istio-podinfo\") pod \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") "
Oct 30 17:13:15.095704 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.095686    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a82e355-1770-48e7-a8ab-92d5d1b6d257-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a82e355-1770-48e7-a8ab-92d5d1b6d257-istio-data\") pod \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") "
Oct 30 17:13:15.095875 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.095849    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a82e355-1770-48e7-a8ab-92d5d1b6d257-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a82e355-1770-48e7-a8ab-92d5d1b6d257-workload-certs\") pod \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") "
Oct 30 17:13:15.095956 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.095910    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a82e355-1770-48e7-a8ab-92d5d1b6d257-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a82e355-1770-48e7-a8ab-92d5d1b6d257-credential-socket\") pod \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\" (UID: \"1a82e355-1770-48e7-a8ab-92d5d1b6d257\") "
Oct 30 17:13:15.096137 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.095990    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a82e355-1770-48e7-a8ab-92d5d1b6d257-workload-socket" pod "1a82e355-1770-48e7-a8ab-92d5d1b6d257" (UID: "1a82e355-1770-48e7-a8ab-92d5d1b6d257"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:13:15.096598 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.096345    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a82e355-1770-48e7-a8ab-92d5d1b6d257-istiod-ca-cert" pod "1a82e355-1770-48e7-a8ab-92d5d1b6d257" (UID: "1a82e355-1770-48e7-a8ab-92d5d1b6d257"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:13:15.096710 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.096629    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a82e355-1770-48e7-a8ab-92d5d1b6d257-credential-socket" pod "1a82e355-1770-48e7-a8ab-92d5d1b6d257" (UID: "1a82e355-1770-48e7-a8ab-92d5d1b6d257"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:13:15.096814 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.096794    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a82e355-1770-48e7-a8ab-92d5d1b6d257-istio-data" pod "1a82e355-1770-48e7-a8ab-92d5d1b6d257" (UID: "1a82e355-1770-48e7-a8ab-92d5d1b6d257"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:13:15.096944 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.096911    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a82e355-1770-48e7-a8ab-92d5d1b6d257-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:15.097031 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.096958    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a82e355-1770-48e7-a8ab-92d5d1b6d257-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:15.097031 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.096974    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a82e355-1770-48e7-a8ab-92d5d1b6d257-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:15.097182 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.097164    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a82e355-1770-48e7-a8ab-92d5d1b6d257-workload-certs" pod "1a82e355-1770-48e7-a8ab-92d5d1b6d257" (UID: "1a82e355-1770-48e7-a8ab-92d5d1b6d257"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:13:15.100272 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.100252    4433 scope.go:122] "RemoveContainer" containerID="b462e2478aa9042db5e06f2b1a383ea7f4f79460671163bb4104fdd809203b85"
Oct 30 17:13:15.100645 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:13:15.100610    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b462e2478aa9042db5e06f2b1a383ea7f4f79460671163bb4104fdd809203b85\": container with ID starting with b462e2478aa9042db5e06f2b1a383ea7f4f79460671163bb4104fdd809203b85 not found: ID does not exist" containerID="b462e2478aa9042db5e06f2b1a383ea7f4f79460671163bb4104fdd809203b85"
Oct 30 17:13:15.100744 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.100649    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b462e2478aa9042db5e06f2b1a383ea7f4f79460671163bb4104fdd809203b85"} err="failed to get container status \"b462e2478aa9042db5e06f2b1a383ea7f4f79460671163bb4104fdd809203b85\": rpc error: code = NotFound desc = could not find container \"b462e2478aa9042db5e06f2b1a383ea7f4f79460671163bb4104fdd809203b85\": container with ID starting with b462e2478aa9042db5e06f2b1a383ea7f4f79460671163bb4104fdd809203b85 not found: ID does not exist"
Oct 30 17:13:15.100744 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.100671    4433 scope.go:122] "RemoveContainer" containerID="adc5cfb160ede6a8214cee4f1864243b93292e6761062e1809b31e6367de7ed2"
Oct 30 17:13:15.101001 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.100978    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a82e355-1770-48e7-a8ab-92d5d1b6d257-kube-api-access-tj6rj" pod "1a82e355-1770-48e7-a8ab-92d5d1b6d257" (UID: "1a82e355-1770-48e7-a8ab-92d5d1b6d257"). InnerVolumeSpecName "kube-api-access-tj6rj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:13:15.101082 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.100981    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a82e355-1770-48e7-a8ab-92d5d1b6d257-istio-envoy" pod "1a82e355-1770-48e7-a8ab-92d5d1b6d257" (UID: "1a82e355-1770-48e7-a8ab-92d5d1b6d257"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:13:15.103545 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.102972    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a82e355-1770-48e7-a8ab-92d5d1b6d257-istio-podinfo" pod "1a82e355-1770-48e7-a8ab-92d5d1b6d257" (UID: "1a82e355-1770-48e7-a8ab-92d5d1b6d257"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:13:15.103545 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.103016    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a82e355-1770-48e7-a8ab-92d5d1b6d257-istio-token" pod "1a82e355-1770-48e7-a8ab-92d5d1b6d257" (UID: "1a82e355-1770-48e7-a8ab-92d5d1b6d257"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:13:15.108017 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.107997    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"
Oct 30 17:13:15.111629 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.111610    4433 scope.go:122] "RemoveContainer" containerID="adc5cfb160ede6a8214cee4f1864243b93292e6761062e1809b31e6367de7ed2"
Oct 30 17:13:15.111939 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:13:15.111918    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adc5cfb160ede6a8214cee4f1864243b93292e6761062e1809b31e6367de7ed2\": container with ID starting with adc5cfb160ede6a8214cee4f1864243b93292e6761062e1809b31e6367de7ed2 not found: ID does not exist" containerID="adc5cfb160ede6a8214cee4f1864243b93292e6761062e1809b31e6367de7ed2"
Oct 30 17:13:15.111999 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.111946    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adc5cfb160ede6a8214cee4f1864243b93292e6761062e1809b31e6367de7ed2"} err="failed to get container status \"adc5cfb160ede6a8214cee4f1864243b93292e6761062e1809b31e6367de7ed2\": rpc error: code = NotFound desc = could not find container \"adc5cfb160ede6a8214cee4f1864243b93292e6761062e1809b31e6367de7ed2\": container with ID starting with adc5cfb160ede6a8214cee4f1864243b93292e6761062e1809b31e6367de7ed2 not found: ID does not exist"
Oct 30 17:13:15.111999 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.111963    4433 scope.go:122] "RemoveContainer" containerID="f15641e0650eb18d0e9ac76f8e575f111731df480cebe92de1b855a61cab3a82"
Oct 30 17:13:15.122684 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.122665    4433 scope.go:122] "RemoveContainer" containerID="f15641e0650eb18d0e9ac76f8e575f111731df480cebe92de1b855a61cab3a82"
Oct 30 17:13:15.122965 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:13:15.122947    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f15641e0650eb18d0e9ac76f8e575f111731df480cebe92de1b855a61cab3a82\": container with ID starting with f15641e0650eb18d0e9ac76f8e575f111731df480cebe92de1b855a61cab3a82 not found: ID does not exist" containerID="f15641e0650eb18d0e9ac76f8e575f111731df480cebe92de1b855a61cab3a82"
Oct 30 17:13:15.123024 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.122971    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f15641e0650eb18d0e9ac76f8e575f111731df480cebe92de1b855a61cab3a82"} err="failed to get container status \"f15641e0650eb18d0e9ac76f8e575f111731df480cebe92de1b855a61cab3a82\": rpc error: code = NotFound desc = could not find container \"f15641e0650eb18d0e9ac76f8e575f111731df480cebe92de1b855a61cab3a82\": container with ID starting with f15641e0650eb18d0e9ac76f8e575f111731df480cebe92de1b855a61cab3a82 not found: ID does not exist"
Oct 30 17:13:15.169863 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.169825    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"]
Oct 30 17:13:15.174169 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.174145    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bggdz"]
Oct 30 17:13:15.198313 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.198282    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-istio-podinfo\") pod \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") "
Oct 30 17:13:15.198504 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.198350    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-credential-socket\") pod \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") "
Oct 30 17:13:15.198504 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.198389    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-kube-api-access-zp549\" (UniqueName: \"kubernetes.io/projected/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-kube-api-access-zp549\") pod \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") "
Oct 30 17:13:15.198504 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.198455    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-istiod-ca-cert\") pod \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") "
Oct 30 17:13:15.198504 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.198480    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-istio-data\") pod \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") "
Oct 30 17:13:15.198733 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.198524    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-workload-certs\") pod \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") "
Oct 30 17:13:15.198733 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.198637    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-istio-envoy\") pod \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") "
Oct 30 17:13:15.198733 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.198727    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-workload-socket\") pod \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") "
Oct 30 17:13:15.198893 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.198772    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-istio-token\" (UniqueName: \"kubernetes.io/projected/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-istio-token\") pod \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\" (UID: \"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e\") "
Oct 30 17:13:15.199239 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.199042    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-credential-socket" pod "20dd24a1-9e9c-40d2-8ad8-bf0a206f292e" (UID: "20dd24a1-9e9c-40d2-8ad8-bf0a206f292e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:13:15.199239 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.199059    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-istiod-ca-cert" pod "20dd24a1-9e9c-40d2-8ad8-bf0a206f292e" (UID: "20dd24a1-9e9c-40d2-8ad8-bf0a206f292e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:13:15.199420 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.199294    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-workload-socket" pod "20dd24a1-9e9c-40d2-8ad8-bf0a206f292e" (UID: "20dd24a1-9e9c-40d2-8ad8-bf0a206f292e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:13:15.200190 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.199646    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-workload-certs" pod "20dd24a1-9e9c-40d2-8ad8-bf0a206f292e" (UID: "20dd24a1-9e9c-40d2-8ad8-bf0a206f292e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:13:15.200190 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.199913    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:15.200190 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.199937    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a82e355-1770-48e7-a8ab-92d5d1b6d257-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:15.200190 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.199954    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:15.200190 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.199967    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:15.200190 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.199980    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a82e355-1770-48e7-a8ab-92d5d1b6d257-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:15.200190 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.199993    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj6rj\" (UniqueName: \"kubernetes.io/projected/1a82e355-1770-48e7-a8ab-92d5d1b6d257-kube-api-access-tj6rj\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:15.200190 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.200010    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a82e355-1770-48e7-a8ab-92d5d1b6d257-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:15.200190 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.200024    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a82e355-1770-48e7-a8ab-92d5d1b6d257-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:15.200190 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.200036    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:15.200190 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.200049    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a82e355-1770-48e7-a8ab-92d5d1b6d257-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:15.200190 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.200115    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-istio-data" pod "20dd24a1-9e9c-40d2-8ad8-bf0a206f292e" (UID: "20dd24a1-9e9c-40d2-8ad8-bf0a206f292e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:13:15.201000 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.200979    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-istio-podinfo" pod "20dd24a1-9e9c-40d2-8ad8-bf0a206f292e" (UID: "20dd24a1-9e9c-40d2-8ad8-bf0a206f292e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:13:15.201394 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.201366    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-istio-envoy" pod "20dd24a1-9e9c-40d2-8ad8-bf0a206f292e" (UID: "20dd24a1-9e9c-40d2-8ad8-bf0a206f292e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:13:15.201584 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.201563    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-istio-token" pod "20dd24a1-9e9c-40d2-8ad8-bf0a206f292e" (UID: "20dd24a1-9e9c-40d2-8ad8-bf0a206f292e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:13:15.202804 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.202780    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-kube-api-access-zp549" pod "20dd24a1-9e9c-40d2-8ad8-bf0a206f292e" (UID: "20dd24a1-9e9c-40d2-8ad8-bf0a206f292e"). InnerVolumeSpecName "kube-api-access-zp549". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:13:15.208534 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.208510    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"]
Oct 30 17:13:15.214687 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.214664    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-mzcbm"]
Oct 30 17:13:15.302463 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.302378    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp549\" (UniqueName: \"kubernetes.io/projected/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-kube-api-access-zp549\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:15.302463 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.302458    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:15.302643 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.302477    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:15.302643 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.302490    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:15.302643 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.302504    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:15.444748 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.444713    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"]
Oct 30 17:13:15.448798 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.448772    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xpxtw"]
Oct 30 17:13:15.696192 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.696150    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a82e355-1770-48e7-a8ab-92d5d1b6d257" path="/var/lib/kubelet/pods/1a82e355-1770-48e7-a8ab-92d5d1b6d257/volumes"
Oct 30 17:13:15.696805 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.696780    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b04a22a-0651-49fe-9d44-7c0067c2ad14" path="/var/lib/kubelet/pods/5b04a22a-0651-49fe-9d44-7c0067c2ad14/volumes"
Oct 30 17:13:15.697334 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:15.697312    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6431be3-b51d-40bc-bc87-ddf189905e14" path="/var/lib/kubelet/pods/e6431be3-b51d-40bc-bc87-ddf189905e14/volumes"
Oct 30 17:13:16.101563 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:16.101447    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"
Oct 30 17:13:16.101563 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:16.101456    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479" event={"ID":"20dd24a1-9e9c-40d2-8ad8-bf0a206f292e","Type":"ContainerDied","Data":"79a8f5d435ce1cae0a03b630f8d9a272322959768ac43cba897cb2a5d35c610e"}
Oct 30 17:13:16.101563 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:16.101525    4433 scope.go:122] "RemoveContainer" containerID="50f27108e1e0a0f9bf20bcfd12c55313d37647207fa3d6d2c4714cac324afed8"
Oct 30 17:13:16.154380 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:16.154350    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"]
Oct 30 17:13:16.165087 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:16.165055    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-np479"]
Oct 30 17:13:17.709166 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:17.709125    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20dd24a1-9e9c-40d2-8ad8-bf0a206f292e" path="/var/lib/kubelet/pods/20dd24a1-9e9c-40d2-8ad8-bf0a206f292e/volumes"
Oct 30 17:13:20.915432 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:20.908057    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg" podUID="d32a2aca-cfad-4b04-a149-0e0589dcfa88" containerName="main" containerID="cri-o://acf77e3620e6321040373b19528930dfaf1aae1c51293c26accf68a9b040a7d8" gracePeriod=45
Oct 30 17:13:21.145671 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:21.145629    4433 generic.go:356] "Generic (PLEG): container finished" podID="d32a2aca-cfad-4b04-a149-0e0589dcfa88" containerID="acf77e3620e6321040373b19528930dfaf1aae1c51293c26accf68a9b040a7d8" exitCode=0
Oct 30 17:13:21.145671 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:21.145657    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg" event={"ID":"d32a2aca-cfad-4b04-a149-0e0589dcfa88","Type":"ContainerDied","Data":"acf77e3620e6321040373b19528930dfaf1aae1c51293c26accf68a9b040a7d8"}
Oct 30 17:13:21.338438 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:21.338364    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg" podUID="d32a2aca-cfad-4b04-a149-0e0589dcfa88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": dial tcp 10.133.0.97:8000: connect: connection refused"
Oct 30 17:13:21.764657 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:21.764631    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"
Oct 30 17:13:21.828617 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:21.828580    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d32a2aca-cfad-4b04-a149-0e0589dcfa88-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d32a2aca-cfad-4b04-a149-0e0589dcfa88-model-cache\") pod \"d32a2aca-cfad-4b04-a149-0e0589dcfa88\" (UID: \"d32a2aca-cfad-4b04-a149-0e0589dcfa88\") "
Oct 30 17:13:21.828896 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:21.828718    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d32a2aca-cfad-4b04-a149-0e0589dcfa88-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d32a2aca-cfad-4b04-a149-0e0589dcfa88-tmp-dir\") pod \"d32a2aca-cfad-4b04-a149-0e0589dcfa88\" (UID: \"d32a2aca-cfad-4b04-a149-0e0589dcfa88\") "
Oct 30 17:13:21.828896 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:21.828840    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d32a2aca-cfad-4b04-a149-0e0589dcfa88-tls-certs\" (UniqueName: \"kubernetes.io/secret/d32a2aca-cfad-4b04-a149-0e0589dcfa88-tls-certs\") pod \"d32a2aca-cfad-4b04-a149-0e0589dcfa88\" (UID: \"d32a2aca-cfad-4b04-a149-0e0589dcfa88\") "
Oct 30 17:13:21.828896 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:21.828869    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d32a2aca-cfad-4b04-a149-0e0589dcfa88-home\" (UniqueName: \"kubernetes.io/empty-dir/d32a2aca-cfad-4b04-a149-0e0589dcfa88-home\") pod \"d32a2aca-cfad-4b04-a149-0e0589dcfa88\" (UID: \"d32a2aca-cfad-4b04-a149-0e0589dcfa88\") "
Oct 30 17:13:21.829030 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:21.828952    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d32a2aca-cfad-4b04-a149-0e0589dcfa88-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d32a2aca-cfad-4b04-a149-0e0589dcfa88-dshm\") pod \"d32a2aca-cfad-4b04-a149-0e0589dcfa88\" (UID: \"d32a2aca-cfad-4b04-a149-0e0589dcfa88\") "
Oct 30 17:13:21.829030 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:21.829006    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d32a2aca-cfad-4b04-a149-0e0589dcfa88-kube-api-access-rwmxz\" (UniqueName: \"kubernetes.io/projected/d32a2aca-cfad-4b04-a149-0e0589dcfa88-kube-api-access-rwmxz\") pod \"d32a2aca-cfad-4b04-a149-0e0589dcfa88\" (UID: \"d32a2aca-cfad-4b04-a149-0e0589dcfa88\") "
Oct 30 17:13:21.829459 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:21.829432    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d32a2aca-cfad-4b04-a149-0e0589dcfa88-model-cache" pod "d32a2aca-cfad-4b04-a149-0e0589dcfa88" (UID: "d32a2aca-cfad-4b04-a149-0e0589dcfa88"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:13:21.829459 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:21.829442    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d32a2aca-cfad-4b04-a149-0e0589dcfa88-home" pod "d32a2aca-cfad-4b04-a149-0e0589dcfa88" (UID: "d32a2aca-cfad-4b04-a149-0e0589dcfa88"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:13:21.829830 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:21.829802    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d32a2aca-cfad-4b04-a149-0e0589dcfa88-tmp-dir" pod "d32a2aca-cfad-4b04-a149-0e0589dcfa88" (UID: "d32a2aca-cfad-4b04-a149-0e0589dcfa88"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:13:21.829948 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:21.829809    4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d32a2aca-cfad-4b04-a149-0e0589dcfa88-model-cache\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:21.829948 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:21.829859    4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d32a2aca-cfad-4b04-a149-0e0589dcfa88-home\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:21.835844 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:21.835799    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d32a2aca-cfad-4b04-a149-0e0589dcfa88-tls-certs" pod "d32a2aca-cfad-4b04-a149-0e0589dcfa88" (UID: "d32a2aca-cfad-4b04-a149-0e0589dcfa88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:13:21.835844 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:21.835809    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d32a2aca-cfad-4b04-a149-0e0589dcfa88-kube-api-access-rwmxz" pod "d32a2aca-cfad-4b04-a149-0e0589dcfa88" (UID: "d32a2aca-cfad-4b04-a149-0e0589dcfa88"). InnerVolumeSpecName "kube-api-access-rwmxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:13:21.837214 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:21.837193    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d32a2aca-cfad-4b04-a149-0e0589dcfa88-dshm" pod "d32a2aca-cfad-4b04-a149-0e0589dcfa88" (UID: "d32a2aca-cfad-4b04-a149-0e0589dcfa88"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:13:21.932908 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:21.932787    4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d32a2aca-cfad-4b04-a149-0e0589dcfa88-dshm\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:21.932908 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:21.932836    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwmxz\" (UniqueName: \"kubernetes.io/projected/d32a2aca-cfad-4b04-a149-0e0589dcfa88-kube-api-access-rwmxz\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:21.932908 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:21.932853    4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d32a2aca-cfad-4b04-a149-0e0589dcfa88-tmp-dir\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:21.932908 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:21.932867    4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d32a2aca-cfad-4b04-a149-0e0589dcfa88-tls-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:13:22.166000 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:22.165959    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"
Oct 30 17:13:22.166184 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:22.166024    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg" event={"ID":"d32a2aca-cfad-4b04-a149-0e0589dcfa88","Type":"ContainerDied","Data":"67b18bc167f391f711967b99af24d3c8dcfa3a24b6cda679d46684cdba055295"}
Oct 30 17:13:22.166184 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:22.166091    4433 scope.go:122] "RemoveContainer" containerID="acf77e3620e6321040373b19528930dfaf1aae1c51293c26accf68a9b040a7d8"
Oct 30 17:13:22.223198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:22.223109    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"]
Oct 30 17:13:22.228443 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:22.228393    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-7ebeb6e1/e2e-lifecycle-model-7ebeb6e1-kserve-54f96bc557-hfmbg"]
Oct 30 17:13:23.758441 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:23.754863    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d32a2aca-cfad-4b04-a149-0e0589dcfa88" path="/var/lib/kubelet/pods/d32a2aca-cfad-4b04-a149-0e0589dcfa88/volumes"
Oct 30 17:13:24.657613 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:24.657550    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" podUID="458eba2e-eb03-452a-bcf4-a5e91da83acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:13:24.690503 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:24.685778    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:13:24.768550 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:24.768502    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:13:24.789158 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:24.788759    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:13:24.893439 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:24.892668    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:13:25.124542 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:25.124057    4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:13:25.211545 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:25.211496    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm" podUID="bf43bf3d-f126-40a1-a24f-10214a85ceb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:13:25.380482 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:25.360134    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:13:25.514463 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:25.504418    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:13:25.687492 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:25.672915    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" podUID="458eba2e-eb03-452a-bcf4-a5e91da83acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 17:13:25.687492 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:25.673858    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:13:25.700434 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:25.696775    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:13:25.700434 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:25.697504    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 17:13:25.829426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:25.828596    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:13:25.829426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:25.829234    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 17:13:25.852431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:25.830217    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 17:13:25.852431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:25.830566    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:13:25.897541 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:25.897492    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 17:13:25.907426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:25.898555    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 17:13:26.066087 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:26.037462    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:13:26.066087 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:26.049936    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 17:13:26.274742 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:26.273515    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm" podUID="bf43bf3d-f126-40a1-a24f-10214a85ceb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 30 17:13:26.275218 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:26.275194    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 17:13:26.275353 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:26.275257    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 17:13:26.528572 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:26.528306    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:13:26.529689 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:26.529657    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:13:26.579828 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:26.579797    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:13:26.626725 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:26.621702    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 17:13:26.657630 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:26.657596    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 17:13:26.855258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:26.853789    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" podUID="458eba2e-eb03-452a-bcf4-a5e91da83acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:13:26.870257 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:26.870230    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:13:27.046771 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:13:27.046713    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:13:27.255028 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:13:27.254990    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:13:57.001573 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:13:57.001520    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:14:05.049775 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:05.049741    4433 scope.go:122] "RemoveContainer" containerID="922d88a82ca9465c0c8088ffefe65efeef4bd4f47b648b551985ae426fd0c640"
Oct 30 17:14:10.130415 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:10.130365    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:14:14.080124 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:14.080070    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:14:14.302192 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:14.301515    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:14:14.612590 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:14.612533    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:14:14.643555 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:14.643511    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm" podUID="bf43bf3d-f126-40a1-a24f-10214a85ceb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:14:14.723602 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:14.723541    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:14:14.881605 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:14.881493    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" podUID="458eba2e-eb03-452a-bcf4-a5e91da83acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:14:15.009897 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:15.009842    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:14:15.142489 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:15.141629    4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:14:15.348429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:15.342577    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:14:26.986189 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:14:26.986144    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:14:27.345631 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:27.345541    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"]
Oct 30 17:14:27.345839 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:27.345762    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf" podUID="aab3820c-99d0-43f4-8dc6-f65893969990" containerName="istio-proxy" containerID="cri-o://c4234c955cbd38fa9b30d0010f78d218ba4b852c974192b3a80b70e39f7b0197" gracePeriod=30
Oct 30 17:14:32.641995 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:32.641954    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"
Oct 30 17:14:32.681705 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:32.681668    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab3820c-99d0-43f4-8dc6-f65893969990-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aab3820c-99d0-43f4-8dc6-f65893969990-istio-envoy\") pod \"aab3820c-99d0-43f4-8dc6-f65893969990\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") "
Oct 30 17:14:32.681705 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:32.681710    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab3820c-99d0-43f4-8dc6-f65893969990-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab3820c-99d0-43f4-8dc6-f65893969990-workload-socket\") pod \"aab3820c-99d0-43f4-8dc6-f65893969990\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") "
Oct 30 17:14:32.681950 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:32.681728    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab3820c-99d0-43f4-8dc6-f65893969990-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab3820c-99d0-43f4-8dc6-f65893969990-credential-socket\") pod \"aab3820c-99d0-43f4-8dc6-f65893969990\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") "
Oct 30 17:14:32.681950 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:32.681883    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aab3820c-99d0-43f4-8dc6-f65893969990-kube-api-access-pzn6j\" (UniqueName: \"kubernetes.io/projected/aab3820c-99d0-43f4-8dc6-f65893969990-kube-api-access-pzn6j\") pod \"aab3820c-99d0-43f4-8dc6-f65893969990\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") "
Oct 30 17:14:32.682067 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:32.681970    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aab3820c-99d0-43f4-8dc6-f65893969990-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aab3820c-99d0-43f4-8dc6-f65893969990-istiod-ca-cert\") pod \"aab3820c-99d0-43f4-8dc6-f65893969990\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") "
Oct 30 17:14:32.682067 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:32.682038    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab3820c-99d0-43f4-8dc6-f65893969990-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aab3820c-99d0-43f4-8dc6-f65893969990-istio-data\") pod \"aab3820c-99d0-43f4-8dc6-f65893969990\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") "
Oct 30 17:14:32.682067 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:32.682035    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab3820c-99d0-43f4-8dc6-f65893969990-workload-socket" pod "aab3820c-99d0-43f4-8dc6-f65893969990" (UID: "aab3820c-99d0-43f4-8dc6-f65893969990"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:14:32.682230 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:32.682110    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab3820c-99d0-43f4-8dc6-f65893969990-credential-socket" pod "aab3820c-99d0-43f4-8dc6-f65893969990" (UID: "aab3820c-99d0-43f4-8dc6-f65893969990"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:14:32.682230 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:32.682167    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aab3820c-99d0-43f4-8dc6-f65893969990-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aab3820c-99d0-43f4-8dc6-f65893969990-istio-podinfo\") pod \"aab3820c-99d0-43f4-8dc6-f65893969990\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") "
Oct 30 17:14:32.682230 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:32.682200    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab3820c-99d0-43f4-8dc6-f65893969990-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aab3820c-99d0-43f4-8dc6-f65893969990-workload-certs\") pod \"aab3820c-99d0-43f4-8dc6-f65893969990\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") "
Oct 30 17:14:32.682438 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:32.682360    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab3820c-99d0-43f4-8dc6-f65893969990-istio-data" pod "aab3820c-99d0-43f4-8dc6-f65893969990" (UID: "aab3820c-99d0-43f4-8dc6-f65893969990"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:14:32.682515 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:32.682418    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aab3820c-99d0-43f4-8dc6-f65893969990-istiod-ca-cert" pod "aab3820c-99d0-43f4-8dc6-f65893969990" (UID: "aab3820c-99d0-43f4-8dc6-f65893969990"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:14:32.682593 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:32.682568    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab3820c-99d0-43f4-8dc6-f65893969990-workload-certs" pod "aab3820c-99d0-43f4-8dc6-f65893969990" (UID: "aab3820c-99d0-43f4-8dc6-f65893969990"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:14:32.682650 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:32.682254    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aab3820c-99d0-43f4-8dc6-f65893969990-istio-token\" (UniqueName: \"kubernetes.io/projected/aab3820c-99d0-43f4-8dc6-f65893969990-istio-token\") pod \"aab3820c-99d0-43f4-8dc6-f65893969990\" (UID: \"aab3820c-99d0-43f4-8dc6-f65893969990\") "
Oct 30 17:14:32.683759 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:32.683362    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aab3820c-99d0-43f4-8dc6-f65893969990-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:14:32.683759 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:32.683388    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aab3820c-99d0-43f4-8dc6-f65893969990-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:14:32.683759 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:32.683419    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aab3820c-99d0-43f4-8dc6-f65893969990-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:14:32.683759 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:32.683434    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab3820c-99d0-43f4-8dc6-f65893969990-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:14:32.683759 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:32.683447    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab3820c-99d0-43f4-8dc6-f65893969990-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:14:32.684534 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:32.684491    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab3820c-99d0-43f4-8dc6-f65893969990-istio-envoy" pod "aab3820c-99d0-43f4-8dc6-f65893969990" (UID: "aab3820c-99d0-43f4-8dc6-f65893969990"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:14:32.684886 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:32.684863    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aab3820c-99d0-43f4-8dc6-f65893969990-istio-token" pod "aab3820c-99d0-43f4-8dc6-f65893969990" (UID: "aab3820c-99d0-43f4-8dc6-f65893969990"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:14:32.684886 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:32.684842    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aab3820c-99d0-43f4-8dc6-f65893969990-istio-podinfo" pod "aab3820c-99d0-43f4-8dc6-f65893969990" (UID: "aab3820c-99d0-43f4-8dc6-f65893969990"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:14:32.685018 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:32.684913    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aab3820c-99d0-43f4-8dc6-f65893969990-kube-api-access-pzn6j" pod "aab3820c-99d0-43f4-8dc6-f65893969990" (UID: "aab3820c-99d0-43f4-8dc6-f65893969990"). InnerVolumeSpecName "kube-api-access-pzn6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:14:32.785345 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:32.785298    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aab3820c-99d0-43f4-8dc6-f65893969990-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:14:32.785345 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:32.785333    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aab3820c-99d0-43f4-8dc6-f65893969990-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:14:32.785345 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:32.785348    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aab3820c-99d0-43f4-8dc6-f65893969990-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:14:32.785650 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:32.785360    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzn6j\" (UniqueName: \"kubernetes.io/projected/aab3820c-99d0-43f4-8dc6-f65893969990-kube-api-access-pzn6j\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:14:33.322127 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:33.322092    4433 generic.go:356] "Generic (PLEG): container finished" podID="aab3820c-99d0-43f4-8dc6-f65893969990" containerID="c4234c955cbd38fa9b30d0010f78d218ba4b852c974192b3a80b70e39f7b0197" exitCode=0
Oct 30 17:14:33.322350 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:33.322169    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"
Oct 30 17:14:33.322350 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:33.322172    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf" event={"ID":"aab3820c-99d0-43f4-8dc6-f65893969990","Type":"ContainerDied","Data":"c4234c955cbd38fa9b30d0010f78d218ba4b852c974192b3a80b70e39f7b0197"}
Oct 30 17:14:33.322350 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:33.322256    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf" event={"ID":"aab3820c-99d0-43f4-8dc6-f65893969990","Type":"ContainerDied","Data":"30c7d3002f99a9de2076fb1706872bdbc4f8f0ec810316c7dc53466fd6580e44"}
Oct 30 17:14:33.322350 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:33.322279    4433 scope.go:122] "RemoveContainer" containerID="c4234c955cbd38fa9b30d0010f78d218ba4b852c974192b3a80b70e39f7b0197"
Oct 30 17:14:33.334385 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:33.334364    4433 scope.go:122] "RemoveContainer" containerID="c4234c955cbd38fa9b30d0010f78d218ba4b852c974192b3a80b70e39f7b0197"
Oct 30 17:14:33.334735 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:14:33.334716    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4234c955cbd38fa9b30d0010f78d218ba4b852c974192b3a80b70e39f7b0197\": container with ID starting with c4234c955cbd38fa9b30d0010f78d218ba4b852c974192b3a80b70e39f7b0197 not found: ID does not exist" containerID="c4234c955cbd38fa9b30d0010f78d218ba4b852c974192b3a80b70e39f7b0197"
Oct 30 17:14:33.334787 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:33.334748    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4234c955cbd38fa9b30d0010f78d218ba4b852c974192b3a80b70e39f7b0197"} err="failed to get container status \"c4234c955cbd38fa9b30d0010f78d218ba4b852c974192b3a80b70e39f7b0197\": rpc error: code = NotFound desc = could not find container \"c4234c955cbd38fa9b30d0010f78d218ba4b852c974192b3a80b70e39f7b0197\": container with ID starting with c4234c955cbd38fa9b30d0010f78d218ba4b852c974192b3a80b70e39f7b0197 not found: ID does not exist"
Oct 30 17:14:33.421412 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:33.421378    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"]
Oct 30 17:14:33.438678 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:33.438642    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-52lpf"]
Oct 30 17:14:33.697619 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:33.697589    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aab3820c-99d0-43f4-8dc6-f65893969990" path="/var/lib/kubelet/pods/aab3820c-99d0-43f4-8dc6-f65893969990/volumes"
Oct 30 17:14:37.129310 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:37.129272    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-432680-569f95cccc-csjrf"]
Oct 30 17:14:37.129862 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:37.129653    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-432680-569f95cccc-csjrf" podUID="d8c08f00-b142-47ba-a441-6ac129e0f3e8" containerName="maas-api" containerID="cri-o://b1e7f5b6d48b73441e575909cdce41f915d5ffd7951a697a97b80a2256f01e34" gracePeriod=30
Oct 30 17:14:37.439024 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:37.438762    4433 generic.go:356] "Generic (PLEG): container finished" podID="d8c08f00-b142-47ba-a441-6ac129e0f3e8" containerID="b1e7f5b6d48b73441e575909cdce41f915d5ffd7951a697a97b80a2256f01e34" exitCode=0
Oct 30 17:14:37.439024 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:37.438919    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-432680-569f95cccc-csjrf" event={"ID":"d8c08f00-b142-47ba-a441-6ac129e0f3e8","Type":"ContainerDied","Data":"b1e7f5b6d48b73441e575909cdce41f915d5ffd7951a697a97b80a2256f01e34"}
Oct 30 17:14:37.628453 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:37.623619    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-432680-569f95cccc-csjrf"
Oct 30 17:14:37.741492 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:37.740933    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8c08f00-b142-47ba-a441-6ac129e0f3e8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c08f00-b142-47ba-a441-6ac129e0f3e8-openshift-service-ca\") pod \"d8c08f00-b142-47ba-a441-6ac129e0f3e8\" (UID: \"d8c08f00-b142-47ba-a441-6ac129e0f3e8\") "
Oct 30 17:14:37.741492 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:37.741062    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8c08f00-b142-47ba-a441-6ac129e0f3e8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c08f00-b142-47ba-a441-6ac129e0f3e8-metrics-certs\") pod \"d8c08f00-b142-47ba-a441-6ac129e0f3e8\" (UID: \"d8c08f00-b142-47ba-a441-6ac129e0f3e8\") "
Oct 30 17:14:37.741492 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:37.741174    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8c08f00-b142-47ba-a441-6ac129e0f3e8-kube-api-access-cs284\" (UniqueName: \"kubernetes.io/projected/d8c08f00-b142-47ba-a441-6ac129e0f3e8-kube-api-access-cs284\") pod \"d8c08f00-b142-47ba-a441-6ac129e0f3e8\" (UID: \"d8c08f00-b142-47ba-a441-6ac129e0f3e8\") "
Oct 30 17:14:37.741492 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:37.741296    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8c08f00-b142-47ba-a441-6ac129e0f3e8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c08f00-b142-47ba-a441-6ac129e0f3e8-maas-api-tls\") pod \"d8c08f00-b142-47ba-a441-6ac129e0f3e8\" (UID: \"d8c08f00-b142-47ba-a441-6ac129e0f3e8\") "
Oct 30 17:14:37.819682 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:37.784806    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8c08f00-b142-47ba-a441-6ac129e0f3e8-metrics-certs" pod "d8c08f00-b142-47ba-a441-6ac129e0f3e8" (UID: "d8c08f00-b142-47ba-a441-6ac129e0f3e8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:14:37.819682 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:37.791590    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8c08f00-b142-47ba-a441-6ac129e0f3e8-maas-api-tls" pod "d8c08f00-b142-47ba-a441-6ac129e0f3e8" (UID: "d8c08f00-b142-47ba-a441-6ac129e0f3e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:14:37.852807 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:37.852756    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c08f00-b142-47ba-a441-6ac129e0f3e8-maas-api-tls\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:14:37.853082 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:37.853065    4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c08f00-b142-47ba-a441-6ac129e0f3e8-metrics-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:14:37.853796 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:37.853773    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8c08f00-b142-47ba-a441-6ac129e0f3e8-openshift-service-ca" pod "d8c08f00-b142-47ba-a441-6ac129e0f3e8" (UID: "d8c08f00-b142-47ba-a441-6ac129e0f3e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:14:37.857155 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:37.857108    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c08f00-b142-47ba-a441-6ac129e0f3e8-kube-api-access-cs284" pod "d8c08f00-b142-47ba-a441-6ac129e0f3e8" (UID: "d8c08f00-b142-47ba-a441-6ac129e0f3e8"). InnerVolumeSpecName "kube-api-access-cs284". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:14:37.954532 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:37.954500    4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c08f00-b142-47ba-a441-6ac129e0f3e8-openshift-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:14:37.954532 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:37.954531    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs284\" (UniqueName: \"kubernetes.io/projected/d8c08f00-b142-47ba-a441-6ac129e0f3e8-kube-api-access-cs284\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:14:38.446587 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:38.446552    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-432680-569f95cccc-csjrf"
Oct 30 17:14:38.447059 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:38.446580    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-432680-569f95cccc-csjrf" event={"ID":"d8c08f00-b142-47ba-a441-6ac129e0f3e8","Type":"ContainerDied","Data":"c0e69bbe337c7e531aa1ba81e7d6654ab5bfa8d547a61a2063df4f5db33185e5"}
Oct 30 17:14:38.447059 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:38.446658    4433 scope.go:122] "RemoveContainer" containerID="b1e7f5b6d48b73441e575909cdce41f915d5ffd7951a697a97b80a2256f01e34"
Oct 30 17:14:38.498809 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:38.498773    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-432680-569f95cccc-csjrf"]
Oct 30 17:14:38.509663 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:38.509632    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-432680-569f95cccc-csjrf"]
Oct 30 17:14:39.695799 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:39.695767    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8c08f00-b142-47ba-a441-6ac129e0f3e8" path="/var/lib/kubelet/pods/d8c08f00-b142-47ba-a441-6ac129e0f3e8/volumes"
Oct 30 17:14:56.979501 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:14:56.979373    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:14:58.749511 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:58.749457    4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:14:59.049591 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:59.049490    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:14:59.325648 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:14:59.325528    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:15:00.154235 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.154197    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb"]
Oct 30 17:15:00.155047 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.155022    4433 cpu_manager.go:414] "containerMap: removing container" podUID="56a5d068-44be-4afa-8811-b9b89f5dd25e" containerName="payload-pre-processing"
Oct 30 17:15:00.155148 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.155049    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a5d068-44be-4afa-8811-b9b89f5dd25e" containerName="payload-pre-processing"
Oct 30 17:15:00.155148 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.155091    4433 cpu_manager.go:414] "containerMap: removing container" podUID="5b04a22a-0651-49fe-9d44-7c0067c2ad14" containerName="istio-proxy"
Oct 30 17:15:00.155148 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.155099    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b04a22a-0651-49fe-9d44-7c0067c2ad14" containerName="istio-proxy"
Oct 30 17:15:00.155148 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.155112    4433 cpu_manager.go:414] "containerMap: removing container" podUID="83c7d819-6442-4e39-b1cc-bb36c62d5732" containerName="istio-proxy"
Oct 30 17:15:00.155148 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.155120    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83c7d819-6442-4e39-b1cc-bb36c62d5732" containerName="istio-proxy"
Oct 30 17:15:00.155148 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.155132    4433 cpu_manager.go:414] "containerMap: removing container" podUID="d8c08f00-b142-47ba-a441-6ac129e0f3e8" containerName="maas-api"
Oct 30 17:15:00.155148 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.155140    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8c08f00-b142-47ba-a441-6ac129e0f3e8" containerName="maas-api"
Oct 30 17:15:00.155580 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.155151    4433 cpu_manager.go:414] "containerMap: removing container" podUID="20dd24a1-9e9c-40d2-8ad8-bf0a206f292e" containerName="istio-proxy"
Oct 30 17:15:00.155580 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.155158    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20dd24a1-9e9c-40d2-8ad8-bf0a206f292e" containerName="istio-proxy"
Oct 30 17:15:00.155580 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.155173    4433 cpu_manager.go:414] "containerMap: removing container" podUID="1a82e355-1770-48e7-a8ab-92d5d1b6d257" containerName="istio-proxy"
Oct 30 17:15:00.155580 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.155180    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a82e355-1770-48e7-a8ab-92d5d1b6d257" containerName="istio-proxy"
Oct 30 17:15:00.155580 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.155204    4433 cpu_manager.go:414] "containerMap: removing container" podUID="aab3820c-99d0-43f4-8dc6-f65893969990" containerName="istio-proxy"
Oct 30 17:15:00.155580 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.155211    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aab3820c-99d0-43f4-8dc6-f65893969990" containerName="istio-proxy"
Oct 30 17:15:00.155580 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.155222    4433 cpu_manager.go:414] "containerMap: removing container" podUID="e6431be3-b51d-40bc-bc87-ddf189905e14" containerName="istio-proxy"
Oct 30 17:15:00.155580 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.155229    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6431be3-b51d-40bc-bc87-ddf189905e14" containerName="istio-proxy"
Oct 30 17:15:00.155580 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.155249    4433 cpu_manager.go:414] "containerMap: removing container" podUID="5b1905ac-5888-4977-b8df-92959b8c4d67" containerName="payload-pre-processing"
Oct 30 17:15:00.155580 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.155256    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b1905ac-5888-4977-b8df-92959b8c4d67" containerName="payload-pre-processing"
Oct 30 17:15:00.155580 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.155281    4433 cpu_manager.go:414] "containerMap: removing container" podUID="a59fb466-c9be-42d1-8980-8cf2b28d71be" containerName="maas-api"
Oct 30 17:15:00.155580 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.155290    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a59fb466-c9be-42d1-8980-8cf2b28d71be" containerName="maas-api"
Oct 30 17:15:00.155580 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.155303    4433 cpu_manager.go:414] "containerMap: removing container" podUID="d32a2aca-cfad-4b04-a149-0e0589dcfa88" containerName="main"
Oct 30 17:15:00.155580 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.155310    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d32a2aca-cfad-4b04-a149-0e0589dcfa88" containerName="main"
Oct 30 17:15:00.155580 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.155469    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="83c7d819-6442-4e39-b1cc-bb36c62d5732" containerName="istio-proxy"
Oct 30 17:15:00.155580 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.155485    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b1905ac-5888-4977-b8df-92959b8c4d67" containerName="payload-pre-processing"
Oct 30 17:15:00.155580 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.155500    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="a59fb466-c9be-42d1-8980-8cf2b28d71be" containerName="maas-api"
Oct 30 17:15:00.155580 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.155510    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="d32a2aca-cfad-4b04-a149-0e0589dcfa88" containerName="main"
Oct 30 17:15:00.155580 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.155521    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a5d068-44be-4afa-8811-b9b89f5dd25e" containerName="payload-pre-processing"
Oct 30 17:15:00.155580 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.155535    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b04a22a-0651-49fe-9d44-7c0067c2ad14" containerName="istio-proxy"
Oct 30 17:15:00.155580 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.155546    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8c08f00-b142-47ba-a441-6ac129e0f3e8" containerName="maas-api"
Oct 30 17:15:00.155580 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.155555    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="20dd24a1-9e9c-40d2-8ad8-bf0a206f292e" containerName="istio-proxy"
Oct 30 17:15:00.155580 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.155564    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a82e355-1770-48e7-a8ab-92d5d1b6d257" containerName="istio-proxy"
Oct 30 17:15:00.155580 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.155575    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="aab3820c-99d0-43f4-8dc6-f65893969990" containerName="istio-proxy"
Oct 30 17:15:00.155580 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.155583    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6431be3-b51d-40bc-bc87-ddf189905e14" containerName="istio-proxy"
Oct 30 17:15:00.159706 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.159683    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb"
Oct 30 17:15:00.191235 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.191192    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb"]
Oct 30 17:15:00.249435 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.242783    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\""
Oct 30 17:15:00.263180 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.263135    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqxtr\" (UniqueName: \"kubernetes.io/projected/b0ab47a4-a873-43e7-b528-bc733d58e7a7-kube-api-access-tqxtr\") pod \"maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb\" (UID: \"b0ab47a4-a873-43e7-b528-bc733d58e7a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb"
Oct 30 17:15:00.315667 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.315017    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ab47a4-a873-43e7-b528-bc733d58e7a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb\" (UID: \"b0ab47a4-a873-43e7-b528-bc733d58e7a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb"
Oct 30 17:15:00.321355 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.321323    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk"]
Oct 30 17:15:00.322507 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.322464    4433 cpu_manager.go:414] "containerMap: removing container" podUID="83c7d819-6442-4e39-b1cc-bb36c62d5732" containerName="istio-proxy"
Oct 30 17:15:00.322859 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.322841    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83c7d819-6442-4e39-b1cc-bb36c62d5732" containerName="istio-proxy"
Oct 30 17:15:00.323218 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.323202    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="83c7d819-6442-4e39-b1cc-bb36c62d5732" containerName="istio-proxy"
Oct 30 17:15:00.327828 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.327810    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk"
Oct 30 17:15:00.343366 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.343319    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl"]
Oct 30 17:15:00.349349 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.349322    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl"]
Oct 30 17:15:00.350041 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.349532    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl"
Oct 30 17:15:00.354849 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.354824    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl"
Oct 30 17:15:00.355455 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.355434    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk"]
Oct 30 17:15:00.382263 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.382227    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889675-djsxq"]
Oct 30 17:15:00.391506 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.391471    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc"]
Oct 30 17:15:00.392013 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.391989    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889675-djsxq"
Oct 30 17:15:00.400799 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.400050    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl"]
Oct 30 17:15:00.400799 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.400158    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc"
Oct 30 17:15:00.403629 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.403603    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl"]
Oct 30 17:15:00.407684 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.407077    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889675-4tt58"]
Oct 30 17:15:00.414254 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.414202    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr"]
Oct 30 17:15:00.414578 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.414557    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889675-4tt58"
Oct 30 17:15:00.417236 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.417213    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv6rh\" (UniqueName: \"kubernetes.io/projected/4e355962-aa3b-4c35-8f1e-b98f41716880-kube-api-access-kv6rh\") pod \"maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk\" (UID: \"4e355962-aa3b-4c35-8f1e-b98f41716880\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk"
Oct 30 17:15:00.417776 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.417607    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqxtr\" (UniqueName: \"kubernetes.io/projected/b0ab47a4-a873-43e7-b528-bc733d58e7a7-kube-api-access-tqxtr\") pod \"maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb\" (UID: \"b0ab47a4-a873-43e7-b528-bc733d58e7a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb"
Oct 30 17:15:00.417776 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.417652    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ab47a4-a873-43e7-b528-bc733d58e7a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb\" (UID: \"b0ab47a4-a873-43e7-b528-bc733d58e7a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb"
Oct 30 17:15:00.417776 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.417680    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e355962-aa3b-4c35-8f1e-b98f41716880-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk\" (UID: \"4e355962-aa3b-4c35-8f1e-b98f41716880\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk"
Oct 30 17:15:00.418781 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.418445    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ab47a4-a873-43e7-b528-bc733d58e7a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb\" (UID: \"b0ab47a4-a873-43e7-b528-bc733d58e7a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb"
Oct 30 17:15:00.419730 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.419704    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889675-djsxq"]
Oct 30 17:15:00.419835 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.419756    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc"]
Oct 30 17:15:00.419912 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.419886    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr"
Oct 30 17:15:00.423393 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.423371    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889675-4tt58"]
Oct 30 17:15:00.423822 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.423802    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr"]
Oct 30 17:15:00.440766 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.440738    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqxtr\" (UniqueName: \"kubernetes.io/projected/b0ab47a4-a873-43e7-b528-bc733d58e7a7-kube-api-access-tqxtr\") pod \"maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb\" (UID: \"b0ab47a4-a873-43e7-b528-bc733d58e7a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb"
Oct 30 17:15:00.481336 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.481305    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb"
Oct 30 17:15:00.519186 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.519151    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5jmx\" (UniqueName: \"kubernetes.io/projected/de736fdc-db4a-4b35-b623-5a037e0579d6-kube-api-access-k5jmx\") pod \"maas-api-key-cleanup-e2e-worker-w0-096d75-29889675-4tt58\" (UID: \"de736fdc-db4a-4b35-b623-5a037e0579d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889675-4tt58"
Oct 30 17:15:00.519356 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.519199    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzwqn\" (UniqueName: \"kubernetes.io/projected/c34089da-9f1d-461a-a3ff-65b5d894eef4-kube-api-access-pzwqn\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889675-djsxq\" (UID: \"c34089da-9f1d-461a-a3ff-65b5d894eef4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889675-djsxq"
Oct 30 17:15:00.519356 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.519328    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw8vs\" (UniqueName: \"kubernetes.io/projected/b78c3100-b1b6-4204-af12-54870e4b8dbb-kube-api-access-tw8vs\") pod \"maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl\" (UID: \"b78c3100-b1b6-4204-af12-54870e4b8dbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl"
Oct 30 17:15:00.519536 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.519465    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/938091c7-0d73-4944-a33b-f123c256a2a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl\" (UID: \"938091c7-0d73-4944-a33b-f123c256a2a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl"
Oct 30 17:15:00.519536 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.519490    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c34089da-9f1d-461a-a3ff-65b5d894eef4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889675-djsxq\" (UID: \"c34089da-9f1d-461a-a3ff-65b5d894eef4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889675-djsxq"
Oct 30 17:15:00.519536 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.519515    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr\" (UID: \"ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr"
Oct 30 17:15:00.519694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.519545    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rncx2\" (UniqueName: \"kubernetes.io/projected/1aa1ca72-36df-4d56-b3f2-e7b33f4e1379-kube-api-access-rncx2\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc\" (UID: \"1aa1ca72-36df-4d56-b3f2-e7b33f4e1379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc"
Oct 30 17:15:00.519694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.519676    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjkqh\" (UniqueName: \"kubernetes.io/projected/ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b-kube-api-access-cjkqh\") pod \"maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr\" (UID: \"ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr"
Oct 30 17:15:00.519794 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.519715    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b78c3100-b1b6-4204-af12-54870e4b8dbb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl\" (UID: \"b78c3100-b1b6-4204-af12-54870e4b8dbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl"
Oct 30 17:15:00.519866 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.519786    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa1ca72-36df-4d56-b3f2-e7b33f4e1379-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc\" (UID: \"1aa1ca72-36df-4d56-b3f2-e7b33f4e1379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc"
Oct 30 17:15:00.519866 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.519815    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk5mr\" (UniqueName: \"kubernetes.io/projected/938091c7-0d73-4944-a33b-f123c256a2a4-kube-api-access-xk5mr\") pod \"maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl\" (UID: \"938091c7-0d73-4944-a33b-f123c256a2a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl"
Oct 30 17:15:00.519969 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.519949    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e355962-aa3b-4c35-8f1e-b98f41716880-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk\" (UID: \"4e355962-aa3b-4c35-8f1e-b98f41716880\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk"
Oct 30 17:15:00.520026 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.519980    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de736fdc-db4a-4b35-b623-5a037e0579d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-096d75-29889675-4tt58\" (UID: \"de736fdc-db4a-4b35-b623-5a037e0579d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889675-4tt58"
Oct 30 17:15:00.520084 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.520042    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv6rh\" (UniqueName: \"kubernetes.io/projected/4e355962-aa3b-4c35-8f1e-b98f41716880-kube-api-access-kv6rh\") pod \"maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk\" (UID: \"4e355962-aa3b-4c35-8f1e-b98f41716880\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk"
Oct 30 17:15:00.520683 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.520650    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e355962-aa3b-4c35-8f1e-b98f41716880-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk\" (UID: \"4e355962-aa3b-4c35-8f1e-b98f41716880\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk"
Oct 30 17:15:00.543421 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.543382    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv6rh\" (UniqueName: \"kubernetes.io/projected/4e355962-aa3b-4c35-8f1e-b98f41716880-kube-api-access-kv6rh\") pod \"maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk\" (UID: \"4e355962-aa3b-4c35-8f1e-b98f41716880\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk"
Oct 30 17:15:00.621630 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.621585    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de736fdc-db4a-4b35-b623-5a037e0579d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-096d75-29889675-4tt58\" (UID: \"de736fdc-db4a-4b35-b623-5a037e0579d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889675-4tt58"
Oct 30 17:15:00.621800 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.621760    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5jmx\" (UniqueName: \"kubernetes.io/projected/de736fdc-db4a-4b35-b623-5a037e0579d6-kube-api-access-k5jmx\") pod \"maas-api-key-cleanup-e2e-worker-w0-096d75-29889675-4tt58\" (UID: \"de736fdc-db4a-4b35-b623-5a037e0579d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889675-4tt58"
Oct 30 17:15:00.621860 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.621803    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzwqn\" (UniqueName: \"kubernetes.io/projected/c34089da-9f1d-461a-a3ff-65b5d894eef4-kube-api-access-pzwqn\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889675-djsxq\" (UID: \"c34089da-9f1d-461a-a3ff-65b5d894eef4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889675-djsxq"
Oct 30 17:15:00.621860 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.621844    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw8vs\" (UniqueName: \"kubernetes.io/projected/b78c3100-b1b6-4204-af12-54870e4b8dbb-kube-api-access-tw8vs\") pod \"maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl\" (UID: \"b78c3100-b1b6-4204-af12-54870e4b8dbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl"
Oct 30 17:15:00.621954 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.621940    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/938091c7-0d73-4944-a33b-f123c256a2a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl\" (UID: \"938091c7-0d73-4944-a33b-f123c256a2a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl"
Oct 30 17:15:00.621993 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.621968    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c34089da-9f1d-461a-a3ff-65b5d894eef4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889675-djsxq\" (UID: \"c34089da-9f1d-461a-a3ff-65b5d894eef4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889675-djsxq"
Oct 30 17:15:00.622024 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.622002    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr\" (UID: \"ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr"
Oct 30 17:15:00.622068 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.622046    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rncx2\" (UniqueName: \"kubernetes.io/projected/1aa1ca72-36df-4d56-b3f2-e7b33f4e1379-kube-api-access-rncx2\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc\" (UID: \"1aa1ca72-36df-4d56-b3f2-e7b33f4e1379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc"
Oct 30 17:15:00.622106 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.622095    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjkqh\" (UniqueName: \"kubernetes.io/projected/ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b-kube-api-access-cjkqh\") pod \"maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr\" (UID: \"ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr"
Oct 30 17:15:00.622146 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.622127    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b78c3100-b1b6-4204-af12-54870e4b8dbb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl\" (UID: \"b78c3100-b1b6-4204-af12-54870e4b8dbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl"
Oct 30 17:15:00.622186 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.622171    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa1ca72-36df-4d56-b3f2-e7b33f4e1379-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc\" (UID: \"1aa1ca72-36df-4d56-b3f2-e7b33f4e1379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc"
Oct 30 17:15:00.622230 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.622195    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk5mr\" (UniqueName: \"kubernetes.io/projected/938091c7-0d73-4944-a33b-f123c256a2a4-kube-api-access-xk5mr\") pod \"maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl\" (UID: \"938091c7-0d73-4944-a33b-f123c256a2a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl"
Oct 30 17:15:00.622615 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.622506    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de736fdc-db4a-4b35-b623-5a037e0579d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-096d75-29889675-4tt58\" (UID: \"de736fdc-db4a-4b35-b623-5a037e0579d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889675-4tt58"
Oct 30 17:15:00.623312 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.623286    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b78c3100-b1b6-4204-af12-54870e4b8dbb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl\" (UID: \"b78c3100-b1b6-4204-af12-54870e4b8dbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl"
Oct 30 17:15:00.623312 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.623300    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/938091c7-0d73-4944-a33b-f123c256a2a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl\" (UID: \"938091c7-0d73-4944-a33b-f123c256a2a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl"
Oct 30 17:15:00.623491 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.623468    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa1ca72-36df-4d56-b3f2-e7b33f4e1379-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc\" (UID: \"1aa1ca72-36df-4d56-b3f2-e7b33f4e1379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc"
Oct 30 17:15:00.623541 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.623526    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c34089da-9f1d-461a-a3ff-65b5d894eef4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889675-djsxq\" (UID: \"c34089da-9f1d-461a-a3ff-65b5d894eef4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889675-djsxq"
Oct 30 17:15:00.623576 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.623552    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr\" (UID: \"ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr"
Oct 30 17:15:00.645984 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.645946    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzwqn\" (UniqueName: \"kubernetes.io/projected/c34089da-9f1d-461a-a3ff-65b5d894eef4-kube-api-access-pzwqn\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889675-djsxq\" (UID: \"c34089da-9f1d-461a-a3ff-65b5d894eef4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889675-djsxq"
Oct 30 17:15:00.646162 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.646116    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rncx2\" (UniqueName: \"kubernetes.io/projected/1aa1ca72-36df-4d56-b3f2-e7b33f4e1379-kube-api-access-rncx2\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc\" (UID: \"1aa1ca72-36df-4d56-b3f2-e7b33f4e1379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc"
Oct 30 17:15:00.646162 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.646142    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk5mr\" (UniqueName: \"kubernetes.io/projected/938091c7-0d73-4944-a33b-f123c256a2a4-kube-api-access-xk5mr\") pod \"maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl\" (UID: \"938091c7-0d73-4944-a33b-f123c256a2a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl"
Oct 30 17:15:00.653057 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.653027    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5jmx\" (UniqueName: \"kubernetes.io/projected/de736fdc-db4a-4b35-b623-5a037e0579d6-kube-api-access-k5jmx\") pod \"maas-api-key-cleanup-e2e-worker-w0-096d75-29889675-4tt58\" (UID: \"de736fdc-db4a-4b35-b623-5a037e0579d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889675-4tt58"
Oct 30 17:15:00.653211 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.653027    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw8vs\" (UniqueName: \"kubernetes.io/projected/b78c3100-b1b6-4204-af12-54870e4b8dbb-kube-api-access-tw8vs\") pod \"maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl\" (UID: \"b78c3100-b1b6-4204-af12-54870e4b8dbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl"
Oct 30 17:15:00.654078 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.654058    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjkqh\" (UniqueName: \"kubernetes.io/projected/ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b-kube-api-access-cjkqh\") pod \"maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr\" (UID: \"ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr"
Oct 30 17:15:00.657094 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.657073    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk"
Oct 30 17:15:00.697976 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.697940    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl"
Oct 30 17:15:00.704992 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.704842    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl"
Oct 30 17:15:00.720489 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.715674    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889675-djsxq"
Oct 30 17:15:00.724742 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.724572    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc"
Oct 30 17:15:00.740322 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.739246    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889675-4tt58"
Oct 30 17:15:00.747788 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.747707    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr"
Oct 30 17:15:00.801063 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:15:00.801028    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ab47a4_a873_43e7_b528_bc733d58e7a7.slice/crio-cf8dc49948d4b439acda329e86a4c8a3f34943895f8a0020bb3792b8ebded437 WatchSource:0}: Error finding container cf8dc49948d4b439acda329e86a4c8a3f34943895f8a0020bb3792b8ebded437: Status 404 returned error can't find the container with id cf8dc49948d4b439acda329e86a4c8a3f34943895f8a0020bb3792b8ebded437
Oct 30 17:15:00.802546 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.802511    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb"]
Oct 30 17:15:00.941699 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.941660    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb" event={"ID":"b0ab47a4-a873-43e7-b528-bc733d58e7a7","Type":"ContainerStarted","Data":"cf8dc49948d4b439acda329e86a4c8a3f34943895f8a0020bb3792b8ebded437"}
Oct 30 17:15:00.967507 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:00.967480    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk"]
Oct 30 17:15:00.968316 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:15:00.968293    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e355962_aa3b_4c35_8f1e_b98f41716880.slice/crio-e2a0ecfed1fe9ca1c31d855b45d486c70067c37f0369a21e454c815a1925766a WatchSource:0}: Error finding container e2a0ecfed1fe9ca1c31d855b45d486c70067c37f0369a21e454c815a1925766a: Status 404 returned error can't find the container with id e2a0ecfed1fe9ca1c31d855b45d486c70067c37f0369a21e454c815a1925766a
Oct 30 17:15:01.069998 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:01.069968    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl"]
Oct 30 17:15:01.090822 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:01.089340    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl"]
Oct 30 17:15:01.104886 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:01.093635    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889675-djsxq"]
Oct 30 17:15:01.107180 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:15:01.107080    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb78c3100_b1b6_4204_af12_54870e4b8dbb.slice/crio-31f4b375390f6c8b3c3881890c4e3675e89a9d6794f0dc50f605d154b1ab444a WatchSource:0}: Error finding container 31f4b375390f6c8b3c3881890c4e3675e89a9d6794f0dc50f605d154b1ab444a: Status 404 returned error can't find the container with id 31f4b375390f6c8b3c3881890c4e3675e89a9d6794f0dc50f605d154b1ab444a
Oct 30 17:15:01.120383 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:01.120353    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc"]
Oct 30 17:15:01.168820 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:01.167415    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr"]
Oct 30 17:15:01.172953 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:01.172869    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889675-4tt58"]
Oct 30 17:15:01.951968 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:01.951920    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889675-4tt58" event={"ID":"de736fdc-db4a-4b35-b623-5a037e0579d6","Type":"ContainerStarted","Data":"87ee5a295800561fbd1b75ea5f4052678d6cf36af094bca65ec96205a7406594"}
Oct 30 17:15:01.954969 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:01.954939    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb" event={"ID":"b0ab47a4-a873-43e7-b528-bc733d58e7a7","Type":"ContainerStarted","Data":"265a97e43aef5d2df39d6aff4eacea8acbbd876a321dbaeb994c4a027c0b4e81"}
Oct 30 17:15:01.957299 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:01.957272    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc" event={"ID":"1aa1ca72-36df-4d56-b3f2-e7b33f4e1379","Type":"ContainerStarted","Data":"2dce88f81d0b51ecda2e84bca619489aa8c376e87536e0f28e7b6d201af6a0e3"}
Oct 30 17:15:01.957505 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:01.957331    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc" event={"ID":"1aa1ca72-36df-4d56-b3f2-e7b33f4e1379","Type":"ContainerStarted","Data":"a88ab0e9f227d385caf33d9707c49fc749aa5cffc80bfb5ceb98714d45b51e32"}
Oct 30 17:15:01.958681 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:01.958655    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889675-djsxq" event={"ID":"c34089da-9f1d-461a-a3ff-65b5d894eef4","Type":"ContainerStarted","Data":"381456db0737707a95aeeb6cfbb557297b441219cb1fd5bfcd0763174d5691b8"}
Oct 30 17:15:01.960415 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:01.960369    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr" event={"ID":"ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b","Type":"ContainerStarted","Data":"71f743d1b806b950258cba821b13c493e32cde8e26c540f52ea2bfab5a04276e"}
Oct 30 17:15:01.960534 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:01.960445    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr" event={"ID":"ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b","Type":"ContainerStarted","Data":"ab8f4b94f8b3854bf4d1510bb8ed73cd4443dfd1e81e09596d8bc0a2569c5ed1"}
Oct 30 17:15:01.961942 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:01.961922    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl" event={"ID":"b78c3100-b1b6-4204-af12-54870e4b8dbb","Type":"ContainerStarted","Data":"31f4b375390f6c8b3c3881890c4e3675e89a9d6794f0dc50f605d154b1ab444a"}
Oct 30 17:15:01.963931 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:01.963903    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl" event={"ID":"938091c7-0d73-4944-a33b-f123c256a2a4","Type":"ContainerStarted","Data":"85ff6b8e776ef1a3169284fcb824bec6d9c32291203ffc957b084f9d1a5ef7dc"}
Oct 30 17:15:01.964300 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:01.963960    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl" event={"ID":"938091c7-0d73-4944-a33b-f123c256a2a4","Type":"ContainerStarted","Data":"1dc7a5b76e95a120eae2f740d61fb76bb525818188cb09c7965c978c2ccd1867"}
Oct 30 17:15:01.965847 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:01.965825    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk" event={"ID":"4e355962-aa3b-4c35-8f1e-b98f41716880","Type":"ContainerStarted","Data":"0686c91057e53a2000fb2c33f448c79253644d1892169baf6d119340860be206"}
Oct 30 17:15:01.965977 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:01.965958    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk" event={"ID":"4e355962-aa3b-4c35-8f1e-b98f41716880","Type":"ContainerStarted","Data":"e2a0ecfed1fe9ca1c31d855b45d486c70067c37f0369a21e454c815a1925766a"}
Oct 30 17:15:01.999199 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:01.999140    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb" podStartSLOduration=1.999124527 podStartE2EDuration="1.999124527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:01.995678888 +0000 UTC m=+1678.430581094" watchObservedRunningTime="2026-10-30 17:15:01.999124527 +0000 UTC m=+1678.434026732"
Oct 30 17:15:02.035872 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:02.035803    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl" podStartSLOduration=2.035784009 podStartE2EDuration="2.035784009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:02.034323604 +0000 UTC m=+1678.469225811" watchObservedRunningTime="2026-10-30 17:15:02.035784009 +0000 UTC m=+1678.470686215"
Oct 30 17:15:02.083054 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:02.078130    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk" podStartSLOduration=2.078100443 podStartE2EDuration="2.078100443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:02.073213778 +0000 UTC m=+1678.508115986" watchObservedRunningTime="2026-10-30 17:15:02.078100443 +0000 UTC m=+1678.513002654"
Oct 30 17:15:02.115266 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:02.115196    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr" podStartSLOduration=2.115175683 podStartE2EDuration="2.115175683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:02.113559873 +0000 UTC m=+1678.548462071" watchObservedRunningTime="2026-10-30 17:15:02.115175683 +0000 UTC m=+1678.550077890"
Oct 30 17:15:02.975276 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:02.975239    4433 generic.go:356] "Generic (PLEG): container finished" podID="b0ab47a4-a873-43e7-b528-bc733d58e7a7" containerID="265a97e43aef5d2df39d6aff4eacea8acbbd876a321dbaeb994c4a027c0b4e81" exitCode=60
Oct 30 17:15:02.975730 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:02.975324    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb" event={"ID":"b0ab47a4-a873-43e7-b528-bc733d58e7a7","Type":"ContainerDied","Data":"265a97e43aef5d2df39d6aff4eacea8acbbd876a321dbaeb994c4a027c0b4e81"}
Oct 30 17:15:02.975730 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:02.975669    4433 scope.go:122] "RemoveContainer" containerID="265a97e43aef5d2df39d6aff4eacea8acbbd876a321dbaeb994c4a027c0b4e81"
Oct 30 17:15:02.977101 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:02.977076    4433 generic.go:356] "Generic (PLEG): container finished" podID="1aa1ca72-36df-4d56-b3f2-e7b33f4e1379" containerID="2dce88f81d0b51ecda2e84bca619489aa8c376e87536e0f28e7b6d201af6a0e3" exitCode=60
Oct 30 17:15:02.977194 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:02.977158    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc" event={"ID":"1aa1ca72-36df-4d56-b3f2-e7b33f4e1379","Type":"ContainerDied","Data":"2dce88f81d0b51ecda2e84bca619489aa8c376e87536e0f28e7b6d201af6a0e3"}
Oct 30 17:15:02.977446 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:02.977392    4433 scope.go:122] "RemoveContainer" containerID="2dce88f81d0b51ecda2e84bca619489aa8c376e87536e0f28e7b6d201af6a0e3"
Oct 30 17:15:02.978976 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:02.978956    4433 generic.go:356] "Generic (PLEG): container finished" podID="c34089da-9f1d-461a-a3ff-65b5d894eef4" containerID="ae64f2f9ded9c14aaf6ab9c5fe89225796d95de57f1b3f95ef83e7561ad86515" exitCode=60
Oct 30 17:15:02.979066 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:02.979051    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889675-djsxq" event={"ID":"c34089da-9f1d-461a-a3ff-65b5d894eef4","Type":"ContainerDied","Data":"ae64f2f9ded9c14aaf6ab9c5fe89225796d95de57f1b3f95ef83e7561ad86515"}
Oct 30 17:15:02.979341 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:02.979324    4433 scope.go:122] "RemoveContainer" containerID="ae64f2f9ded9c14aaf6ab9c5fe89225796d95de57f1b3f95ef83e7561ad86515"
Oct 30 17:15:02.981095 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:02.980978    4433 generic.go:356] "Generic (PLEG): container finished" podID="ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b" containerID="71f743d1b806b950258cba821b13c493e32cde8e26c540f52ea2bfab5a04276e" exitCode=60
Oct 30 17:15:02.981095 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:02.980999    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr" event={"ID":"ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b","Type":"ContainerDied","Data":"71f743d1b806b950258cba821b13c493e32cde8e26c540f52ea2bfab5a04276e"}
Oct 30 17:15:02.981237 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:02.981221    4433 scope.go:122] "RemoveContainer" containerID="71f743d1b806b950258cba821b13c493e32cde8e26c540f52ea2bfab5a04276e"
Oct 30 17:15:02.982483 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:02.982463    4433 generic.go:356] "Generic (PLEG): container finished" podID="b78c3100-b1b6-4204-af12-54870e4b8dbb" containerID="8a888225bd25812f5526a87840b3f4c04bfc0798fc7bf396aae355615a8ffbf4" exitCode=60
Oct 30 17:15:02.982592 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:02.982545    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl" event={"ID":"b78c3100-b1b6-4204-af12-54870e4b8dbb","Type":"ContainerDied","Data":"8a888225bd25812f5526a87840b3f4c04bfc0798fc7bf396aae355615a8ffbf4"}
Oct 30 17:15:02.982723 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:02.982707    4433 scope.go:122] "RemoveContainer" containerID="8a888225bd25812f5526a87840b3f4c04bfc0798fc7bf396aae355615a8ffbf4"
Oct 30 17:15:02.984071 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:02.984052    4433 generic.go:356] "Generic (PLEG): container finished" podID="938091c7-0d73-4944-a33b-f123c256a2a4" containerID="85ff6b8e776ef1a3169284fcb824bec6d9c32291203ffc957b084f9d1a5ef7dc" exitCode=60
Oct 30 17:15:02.984188 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:02.984135    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl" event={"ID":"938091c7-0d73-4944-a33b-f123c256a2a4","Type":"ContainerDied","Data":"85ff6b8e776ef1a3169284fcb824bec6d9c32291203ffc957b084f9d1a5ef7dc"}
Oct 30 17:15:02.984308 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:02.984293    4433 scope.go:122] "RemoveContainer" containerID="85ff6b8e776ef1a3169284fcb824bec6d9c32291203ffc957b084f9d1a5ef7dc"
Oct 30 17:15:02.985850 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:02.985676    4433 generic.go:356] "Generic (PLEG): container finished" podID="4e355962-aa3b-4c35-8f1e-b98f41716880" containerID="0686c91057e53a2000fb2c33f448c79253644d1892169baf6d119340860be206" exitCode=60
Oct 30 17:15:02.985850 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:02.985801    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk" event={"ID":"4e355962-aa3b-4c35-8f1e-b98f41716880","Type":"ContainerDied","Data":"0686c91057e53a2000fb2c33f448c79253644d1892169baf6d119340860be206"}
Oct 30 17:15:02.986004 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:02.985957    4433 scope.go:122] "RemoveContainer" containerID="0686c91057e53a2000fb2c33f448c79253644d1892169baf6d119340860be206"
Oct 30 17:15:02.989706 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:02.989682    4433 generic.go:356] "Generic (PLEG): container finished" podID="de736fdc-db4a-4b35-b623-5a037e0579d6" containerID="8914e0ac14b7e8bc5153eacbbad14b0c794a9c35f732cdc60a3ce2bab746bee7" exitCode=60
Oct 30 17:15:02.989806 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:02.989744    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889675-4tt58" event={"ID":"de736fdc-db4a-4b35-b623-5a037e0579d6","Type":"ContainerDied","Data":"8914e0ac14b7e8bc5153eacbbad14b0c794a9c35f732cdc60a3ce2bab746bee7"}
Oct 30 17:15:02.990081 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:02.990035    4433 scope.go:122] "RemoveContainer" containerID="8914e0ac14b7e8bc5153eacbbad14b0c794a9c35f732cdc60a3ce2bab746bee7"
Oct 30 17:15:03.044984 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:03.044928    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc" podStartSLOduration=3.044909597 podStartE2EDuration="3.044909597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:02.160829594 +0000 UTC m=+1678.595731800" watchObservedRunningTime="2026-10-30 17:15:03.044909597 +0000 UTC m=+1679.479811806"
Oct 30 17:15:04.004845 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:04.004802    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl" event={"ID":"938091c7-0d73-4944-a33b-f123c256a2a4","Type":"ContainerStarted","Data":"dcf9acaddf523341ad6cb7206beba6d1e9f4f1c10dfe5c952d3459882ed9edd3"}
Oct 30 17:15:04.007529 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:04.007463    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk" event={"ID":"4e355962-aa3b-4c35-8f1e-b98f41716880","Type":"ContainerStarted","Data":"3b83496103b70f05fbdc8e573c0b02f2818783804c286d7930ae055449e99aae"}
Oct 30 17:15:04.010685 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:04.010653    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889675-4tt58" event={"ID":"de736fdc-db4a-4b35-b623-5a037e0579d6","Type":"ContainerStarted","Data":"9289590bd8643ddf4cd8463db57f4a896f7b6d3cbaac8a3645577fb572eff814"}
Oct 30 17:15:04.014537 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:04.014495    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb" event={"ID":"b0ab47a4-a873-43e7-b528-bc733d58e7a7","Type":"ContainerStarted","Data":"ef613ed0a39d0d01b4897644a221158f521ffaa294d28db74c9df65ba02102c1"}
Oct 30 17:15:04.016993 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:04.016967    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc" event={"ID":"1aa1ca72-36df-4d56-b3f2-e7b33f4e1379","Type":"ContainerStarted","Data":"2cc1d1e387947e3a63aecbabb3d927eb9a0da636f0c877ca5b744c4c3af7d9e0"}
Oct 30 17:15:04.019251 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:04.019224    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889675-djsxq" event={"ID":"c34089da-9f1d-461a-a3ff-65b5d894eef4","Type":"ContainerStarted","Data":"49fa54dfb5dfbc5cf7b581760369f4472869899728aa1930bfa7129f94b25430"}
Oct 30 17:15:04.022058 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:04.022037    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr" event={"ID":"ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b","Type":"ContainerStarted","Data":"cb484a471e96e34569aac4184f2dab1b26bdbdadf920e978ea79933d5bd2d078"}
Oct 30 17:15:04.024629 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:04.024607    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl" event={"ID":"b78c3100-b1b6-4204-af12-54870e4b8dbb","Type":"ContainerStarted","Data":"bce633a94fe5755bc049521711cf3387b73a487d56e45992f8ff90fe5a678c3e"}
Oct 30 17:15:04.091339 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:04.091282    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889675-4tt58" podStartSLOduration=4.091267311 podStartE2EDuration="4.091267311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:04.087534098 +0000 UTC m=+1680.522436301" watchObservedRunningTime="2026-10-30 17:15:04.091267311 +0000 UTC m=+1680.526169516"
Oct 30 17:15:04.279973 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:04.268694    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl" podStartSLOduration=4.268658074 podStartE2EDuration="4.268658074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:04.268116374 +0000 UTC m=+1680.703018581" watchObservedRunningTime="2026-10-30 17:15:04.268658074 +0000 UTC m=+1680.703560281"
Oct 30 17:15:04.471350 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:04.471281    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889675-djsxq" podStartSLOduration=4.471261558 podStartE2EDuration="4.471261558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:04.363340032 +0000 UTC m=+1680.798242234" watchObservedRunningTime="2026-10-30 17:15:04.471261558 +0000 UTC m=+1680.906163765"
Oct 30 17:15:05.040648 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:05.040595    4433 generic.go:356] "Generic (PLEG): container finished" podID="938091c7-0d73-4944-a33b-f123c256a2a4" containerID="dcf9acaddf523341ad6cb7206beba6d1e9f4f1c10dfe5c952d3459882ed9edd3" exitCode=60
Oct 30 17:15:05.041168 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:05.040686    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl" event={"ID":"938091c7-0d73-4944-a33b-f123c256a2a4","Type":"ContainerDied","Data":"dcf9acaddf523341ad6cb7206beba6d1e9f4f1c10dfe5c952d3459882ed9edd3"}
Oct 30 17:15:05.041168 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:05.040767    4433 scope.go:122] "RemoveContainer" containerID="85ff6b8e776ef1a3169284fcb824bec6d9c32291203ffc957b084f9d1a5ef7dc"
Oct 30 17:15:05.041296 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:05.041166    4433 scope.go:122] "RemoveContainer" containerID="dcf9acaddf523341ad6cb7206beba6d1e9f4f1c10dfe5c952d3459882ed9edd3"
Oct 30 17:15:05.041611 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:15:05.041581    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl_odh-ai-gateway-infra(938091c7-0d73-4944-a33b-f123c256a2a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl" podUID="938091c7-0d73-4944-a33b-f123c256a2a4"
Oct 30 17:15:05.044070 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:05.044043    4433 generic.go:356] "Generic (PLEG): container finished" podID="4e355962-aa3b-4c35-8f1e-b98f41716880" containerID="3b83496103b70f05fbdc8e573c0b02f2818783804c286d7930ae055449e99aae" exitCode=60
Oct 30 17:15:05.044220 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:05.044129    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk" event={"ID":"4e355962-aa3b-4c35-8f1e-b98f41716880","Type":"ContainerDied","Data":"3b83496103b70f05fbdc8e573c0b02f2818783804c286d7930ae055449e99aae"}
Oct 30 17:15:05.044568 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:05.044540    4433 scope.go:122] "RemoveContainer" containerID="3b83496103b70f05fbdc8e573c0b02f2818783804c286d7930ae055449e99aae"
Oct 30 17:15:05.044978 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:15:05.044919    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk_odh-ai-gateway-infra(4e355962-aa3b-4c35-8f1e-b98f41716880)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk" podUID="4e355962-aa3b-4c35-8f1e-b98f41716880"
Oct 30 17:15:05.046524 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:05.046498    4433 generic.go:356] "Generic (PLEG): container finished" podID="de736fdc-db4a-4b35-b623-5a037e0579d6" containerID="9289590bd8643ddf4cd8463db57f4a896f7b6d3cbaac8a3645577fb572eff814" exitCode=60
Oct 30 17:15:05.046657 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:05.046631    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889675-4tt58" event={"ID":"de736fdc-db4a-4b35-b623-5a037e0579d6","Type":"ContainerDied","Data":"9289590bd8643ddf4cd8463db57f4a896f7b6d3cbaac8a3645577fb572eff814"}
Oct 30 17:15:05.046910 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:05.046891    4433 scope.go:122] "RemoveContainer" containerID="9289590bd8643ddf4cd8463db57f4a896f7b6d3cbaac8a3645577fb572eff814"
Oct 30 17:15:05.047242 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:15:05.047213    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-096d75-29889675-4tt58_odh-ai-gateway-infra(de736fdc-db4a-4b35-b623-5a037e0579d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889675-4tt58" podUID="de736fdc-db4a-4b35-b623-5a037e0579d6"
Oct 30 17:15:05.049484 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:05.049457    4433 generic.go:356] "Generic (PLEG): container finished" podID="b0ab47a4-a873-43e7-b528-bc733d58e7a7" containerID="ef613ed0a39d0d01b4897644a221158f521ffaa294d28db74c9df65ba02102c1" exitCode=60
Oct 30 17:15:05.049600 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:05.049545    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb" event={"ID":"b0ab47a4-a873-43e7-b528-bc733d58e7a7","Type":"ContainerDied","Data":"ef613ed0a39d0d01b4897644a221158f521ffaa294d28db74c9df65ba02102c1"}
Oct 30 17:15:05.049851 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:05.049813    4433 scope.go:122] "RemoveContainer" containerID="ef613ed0a39d0d01b4897644a221158f521ffaa294d28db74c9df65ba02102c1"
Oct 30 17:15:05.050171 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:15:05.050114    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb_odh-ai-gateway-infra(b0ab47a4-a873-43e7-b528-bc733d58e7a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb" podUID="b0ab47a4-a873-43e7-b528-bc733d58e7a7"
Oct 30 17:15:05.055450 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:05.052245    4433 generic.go:356] "Generic (PLEG): container finished" podID="1aa1ca72-36df-4d56-b3f2-e7b33f4e1379" containerID="2cc1d1e387947e3a63aecbabb3d927eb9a0da636f0c877ca5b744c4c3af7d9e0" exitCode=60
Oct 30 17:15:05.055450 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:05.052387    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc" event={"ID":"1aa1ca72-36df-4d56-b3f2-e7b33f4e1379","Type":"ContainerDied","Data":"2cc1d1e387947e3a63aecbabb3d927eb9a0da636f0c877ca5b744c4c3af7d9e0"}
Oct 30 17:15:05.055450 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:05.054070    4433 scope.go:122] "RemoveContainer" containerID="2cc1d1e387947e3a63aecbabb3d927eb9a0da636f0c877ca5b744c4c3af7d9e0"
Oct 30 17:15:05.055450 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:15:05.054421    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc_odh-ai-gateway-infra(1aa1ca72-36df-4d56-b3f2-e7b33f4e1379)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc" podUID="1aa1ca72-36df-4d56-b3f2-e7b33f4e1379"
Oct 30 17:15:05.055450 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:05.055134    4433 generic.go:356] "Generic (PLEG): container finished" podID="c34089da-9f1d-461a-a3ff-65b5d894eef4" containerID="49fa54dfb5dfbc5cf7b581760369f4472869899728aa1930bfa7129f94b25430" exitCode=60
Oct 30 17:15:05.055450 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:05.055223    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889675-djsxq" event={"ID":"c34089da-9f1d-461a-a3ff-65b5d894eef4","Type":"ContainerDied","Data":"49fa54dfb5dfbc5cf7b581760369f4472869899728aa1930bfa7129f94b25430"}
Oct 30 17:15:05.055859 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:05.055475    4433 scope.go:122] "RemoveContainer" containerID="49fa54dfb5dfbc5cf7b581760369f4472869899728aa1930bfa7129f94b25430"
Oct 30 17:15:05.055859 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:15:05.055786    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889675-djsxq_odh-ai-gateway-infra(c34089da-9f1d-461a-a3ff-65b5d894eef4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889675-djsxq" podUID="c34089da-9f1d-461a-a3ff-65b5d894eef4"
Oct 30 17:15:05.057437 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:05.057380    4433 generic.go:356] "Generic (PLEG): container finished" podID="ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b" containerID="cb484a471e96e34569aac4184f2dab1b26bdbdadf920e978ea79933d5bd2d078" exitCode=60
Oct 30 17:15:05.057794 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:05.057773    4433 scope.go:122] "RemoveContainer" containerID="cb484a471e96e34569aac4184f2dab1b26bdbdadf920e978ea79933d5bd2d078"
Oct 30 17:15:05.058042 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:15:05.058019    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr_odh-ai-gateway-infra(ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr" podUID="ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b"
Oct 30 17:15:05.058151 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:05.058063    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr" event={"ID":"ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b","Type":"ContainerDied","Data":"cb484a471e96e34569aac4184f2dab1b26bdbdadf920e978ea79933d5bd2d078"}
Oct 30 17:15:05.060805 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:05.060780    4433 generic.go:356] "Generic (PLEG): container finished" podID="b78c3100-b1b6-4204-af12-54870e4b8dbb" containerID="bce633a94fe5755bc049521711cf3387b73a487d56e45992f8ff90fe5a678c3e" exitCode=60
Oct 30 17:15:05.060946 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:05.060842    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl" event={"ID":"b78c3100-b1b6-4204-af12-54870e4b8dbb","Type":"ContainerDied","Data":"bce633a94fe5755bc049521711cf3387b73a487d56e45992f8ff90fe5a678c3e"}
Oct 30 17:15:05.061196 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:05.061177    4433 scope.go:122] "RemoveContainer" containerID="bce633a94fe5755bc049521711cf3387b73a487d56e45992f8ff90fe5a678c3e"
Oct 30 17:15:05.061499 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:15:05.061474    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl_odh-ai-gateway-infra(b78c3100-b1b6-4204-af12-54870e4b8dbb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl" podUID="b78c3100-b1b6-4204-af12-54870e4b8dbb"
Oct 30 17:15:05.065591 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:05.065565    4433 scope.go:122] "RemoveContainer" containerID="0686c91057e53a2000fb2c33f448c79253644d1892169baf6d119340860be206"
Oct 30 17:15:05.081915 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:05.081891    4433 scope.go:122] "RemoveContainer" containerID="8914e0ac14b7e8bc5153eacbbad14b0c794a9c35f732cdc60a3ce2bab746bee7"
Oct 30 17:15:05.096916 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:05.096893    4433 scope.go:122] "RemoveContainer" containerID="265a97e43aef5d2df39d6aff4eacea8acbbd876a321dbaeb994c4a027c0b4e81"
Oct 30 17:15:05.110827 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:05.110804    4433 scope.go:122] "RemoveContainer" containerID="2dce88f81d0b51ecda2e84bca619489aa8c376e87536e0f28e7b6d201af6a0e3"
Oct 30 17:15:05.129344 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:05.129317    4433 scope.go:122] "RemoveContainer" containerID="ae64f2f9ded9c14aaf6ab9c5fe89225796d95de57f1b3f95ef83e7561ad86515"
Oct 30 17:15:05.149197 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:05.149175    4433 scope.go:122] "RemoveContainer" containerID="71f743d1b806b950258cba821b13c493e32cde8e26c540f52ea2bfab5a04276e"
Oct 30 17:15:05.164057 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:05.164034    4433 scope.go:122] "RemoveContainer" containerID="8a888225bd25812f5526a87840b3f4c04bfc0798fc7bf396aae355615a8ffbf4"
Oct 30 17:15:06.070582 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:06.070555    4433 scope.go:122] "RemoveContainer" containerID="9289590bd8643ddf4cd8463db57f4a896f7b6d3cbaac8a3645577fb572eff814"
Oct 30 17:15:06.071036 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:15:06.070924    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-096d75-29889675-4tt58_odh-ai-gateway-infra(de736fdc-db4a-4b35-b623-5a037e0579d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889675-4tt58" podUID="de736fdc-db4a-4b35-b623-5a037e0579d6"
Oct 30 17:15:06.072381 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:06.072352    4433 scope.go:122] "RemoveContainer" containerID="ef613ed0a39d0d01b4897644a221158f521ffaa294d28db74c9df65ba02102c1"
Oct 30 17:15:06.072974 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:15:06.072720    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb_odh-ai-gateway-infra(b0ab47a4-a873-43e7-b528-bc733d58e7a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb" podUID="b0ab47a4-a873-43e7-b528-bc733d58e7a7"
Oct 30 17:15:06.074159 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:06.074134    4433 scope.go:122] "RemoveContainer" containerID="2cc1d1e387947e3a63aecbabb3d927eb9a0da636f0c877ca5b744c4c3af7d9e0"
Oct 30 17:15:06.074503 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:15:06.074479    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc_odh-ai-gateway-infra(1aa1ca72-36df-4d56-b3f2-e7b33f4e1379)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc" podUID="1aa1ca72-36df-4d56-b3f2-e7b33f4e1379"
Oct 30 17:15:06.076127 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:06.076091    4433 scope.go:122] "RemoveContainer" containerID="49fa54dfb5dfbc5cf7b581760369f4472869899728aa1930bfa7129f94b25430"
Oct 30 17:15:06.076535 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:15:06.076469    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889675-djsxq_odh-ai-gateway-infra(c34089da-9f1d-461a-a3ff-65b5d894eef4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889675-djsxq" podUID="c34089da-9f1d-461a-a3ff-65b5d894eef4"
Oct 30 17:15:06.081805 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:06.079006    4433 scope.go:122] "RemoveContainer" containerID="cb484a471e96e34569aac4184f2dab1b26bdbdadf920e978ea79933d5bd2d078"
Oct 30 17:15:06.081805 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:15:06.079314    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr_odh-ai-gateway-infra(ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr" podUID="ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b"
Oct 30 17:15:06.084731 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:06.084704    4433 scope.go:122] "RemoveContainer" containerID="bce633a94fe5755bc049521711cf3387b73a487d56e45992f8ff90fe5a678c3e"
Oct 30 17:15:06.085070 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:15:06.085049    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl_odh-ai-gateway-infra(b78c3100-b1b6-4204-af12-54870e4b8dbb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl" podUID="b78c3100-b1b6-4204-af12-54870e4b8dbb"
Oct 30 17:15:06.086992 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:06.086970    4433 scope.go:122] "RemoveContainer" containerID="dcf9acaddf523341ad6cb7206beba6d1e9f4f1c10dfe5c952d3459882ed9edd3"
Oct 30 17:15:06.087277 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:15:06.087249    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl_odh-ai-gateway-infra(938091c7-0d73-4944-a33b-f123c256a2a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl" podUID="938091c7-0d73-4944-a33b-f123c256a2a4"
Oct 30 17:15:06.088566 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:06.088544    4433 scope.go:122] "RemoveContainer" containerID="3b83496103b70f05fbdc8e573c0b02f2818783804c286d7930ae055449e99aae"
Oct 30 17:15:06.088808 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:15:06.088786    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk_odh-ai-gateway-infra(4e355962-aa3b-4c35-8f1e-b98f41716880)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk" podUID="4e355962-aa3b-4c35-8f1e-b98f41716880"
Oct 30 17:15:11.903236 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:11.903202    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"]
Oct 30 17:15:11.911531 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:11.911491    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"
Oct 30 17:15:11.945478 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:11.945438    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"]
Oct 30 17:15:11.980800 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:11.980759    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d5mz\" (UniqueName: \"kubernetes.io/projected/e50e437f-d457-49cc-94fc-73aa7340721d-kube-api-access-8d5mz\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"
Oct 30 17:15:11.980800 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:11.980799    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e50e437f-d457-49cc-94fc-73aa7340721d-istio-podinfo\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"
Oct 30 17:15:11.981081 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:11.980822    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e50e437f-d457-49cc-94fc-73aa7340721d-istiod-ca-cert\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"
Oct 30 17:15:11.981081 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:11.980928    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e50e437f-d457-49cc-94fc-73aa7340721d-istio-data\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"
Oct 30 17:15:11.981198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:11.981082    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e50e437f-d457-49cc-94fc-73aa7340721d-istio-token\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"
Oct 30 17:15:11.981198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:11.981112    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50e437f-d457-49cc-94fc-73aa7340721d-credential-socket\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"
Oct 30 17:15:11.981313 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:11.981215    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e50e437f-d457-49cc-94fc-73aa7340721d-workload-certs\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"
Oct 30 17:15:11.981313 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:11.981269    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50e437f-d457-49cc-94fc-73aa7340721d-workload-socket\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"
Oct 30 17:15:11.981313 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:11.981285    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e50e437f-d457-49cc-94fc-73aa7340721d-istio-envoy\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"
Oct 30 17:15:12.011309 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:12.011276    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f42baf-openshift-default-dockercfg-njzvm\""
Oct 30 17:15:12.082659 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:12.082619    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d5mz\" (UniqueName: \"kubernetes.io/projected/e50e437f-d457-49cc-94fc-73aa7340721d-kube-api-access-8d5mz\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"
Oct 30 17:15:12.082659 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:12.082663    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e50e437f-d457-49cc-94fc-73aa7340721d-istio-podinfo\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"
Oct 30 17:15:12.082883 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:12.082683    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e50e437f-d457-49cc-94fc-73aa7340721d-istiod-ca-cert\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"
Oct 30 17:15:12.082883 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:12.082716    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e50e437f-d457-49cc-94fc-73aa7340721d-istio-data\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"
Oct 30 17:15:12.082883 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:12.082850    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e50e437f-d457-49cc-94fc-73aa7340721d-istio-token\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"
Oct 30 17:15:12.082883 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:12.082880    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50e437f-d457-49cc-94fc-73aa7340721d-credential-socket\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"
Oct 30 17:15:12.083076 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:12.082944    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e50e437f-d457-49cc-94fc-73aa7340721d-workload-certs\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"
Oct 30 17:15:12.083076 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:12.083000    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50e437f-d457-49cc-94fc-73aa7340721d-workload-socket\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"
Oct 30 17:15:12.083076 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:12.083021    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e50e437f-d457-49cc-94fc-73aa7340721d-istio-envoy\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"
Oct 30 17:15:12.083277 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:12.083246    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e50e437f-d457-49cc-94fc-73aa7340721d-istio-data\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"
Oct 30 17:15:12.083353 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:12.083315    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50e437f-d457-49cc-94fc-73aa7340721d-credential-socket\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"
Oct 30 17:15:12.083460 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:12.083378    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e50e437f-d457-49cc-94fc-73aa7340721d-workload-certs\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"
Oct 30 17:15:12.083588 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:12.083564    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50e437f-d457-49cc-94fc-73aa7340721d-workload-socket\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"
Oct 30 17:15:12.083659 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:12.083641    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e50e437f-d457-49cc-94fc-73aa7340721d-istiod-ca-cert\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"
Oct 30 17:15:12.085256 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:12.085238    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e50e437f-d457-49cc-94fc-73aa7340721d-istio-envoy\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"
Oct 30 17:15:12.085315 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:12.085297    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e50e437f-d457-49cc-94fc-73aa7340721d-istio-podinfo\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"
Oct 30 17:15:12.107631 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:12.107598    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e50e437f-d457-49cc-94fc-73aa7340721d-istio-token\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"
Oct 30 17:15:12.107764 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:12.107723    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d5mz\" (UniqueName: \"kubernetes.io/projected/e50e437f-d457-49cc-94fc-73aa7340721d-kube-api-access-8d5mz\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"
Oct 30 17:15:12.240139 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:12.240105    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"
Oct 30 17:15:12.765676 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:15:12.765645    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode50e437f_d457_49cc_94fc_73aa7340721d.slice/crio-45426c46aaadc39bb079393ba688fbf93686f12759079b4f97e6b664112f1554 WatchSource:0}: Error finding container 45426c46aaadc39bb079393ba688fbf93686f12759079b4f97e6b664112f1554: Status 404 returned error can't find the container with id 45426c46aaadc39bb079393ba688fbf93686f12759079b4f97e6b664112f1554
Oct 30 17:15:12.766087 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:12.766057    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"]
Oct 30 17:15:12.769061 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:12.769024    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:15:12.769182 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:12.769098    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:15:12.769182 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:12.769141    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:15:13.135686 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:13.135596    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf" event={"ID":"e50e437f-d457-49cc-94fc-73aa7340721d","Type":"ContainerStarted","Data":"cdd767862445376e4927e819feb3ca722099f6b27d26ad271b2f0c9ada783b68"}
Oct 30 17:15:13.135686 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:13.135659    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf" event={"ID":"e50e437f-d457-49cc-94fc-73aa7340721d","Type":"ContainerStarted","Data":"45426c46aaadc39bb079393ba688fbf93686f12759079b4f97e6b664112f1554"}
Oct 30 17:15:13.184771 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:13.184718    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf" podStartSLOduration=2.184699218 podStartE2EDuration="2.184699218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:13.183605921 +0000 UTC m=+1689.618508128" watchObservedRunningTime="2026-10-30 17:15:13.184699218 +0000 UTC m=+1689.619601424"
Oct 30 17:15:13.240296 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:13.240265    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"
Oct 30 17:15:13.242173 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:13.242148    4433 patch_prober.go:29] interesting pod/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 17:15:13.242328 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:13.242197    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf" podUID="e50e437f-d457-49cc-94fc-73aa7340721d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 17:15:14.240567 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:14.240529    4433 patch_prober.go:29] interesting pod/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 17:15:14.241046 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:14.240588    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf" podUID="e50e437f-d457-49cc-94fc-73aa7340721d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 17:15:14.688787 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:14.688756    4433 scope.go:122] "RemoveContainer" containerID="3b83496103b70f05fbdc8e573c0b02f2818783804c286d7930ae055449e99aae"
Oct 30 17:15:15.150293 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:15.150258    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk" event={"ID":"4e355962-aa3b-4c35-8f1e-b98f41716880","Type":"ContainerStarted","Data":"3e22bd7cb34e9f6640d7ab1ff55a4bebc231a3f60aab7db1db0dfc0da5e6cabe"}
Oct 30 17:15:15.240945 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:15.240909    4433 patch_prober.go:29] interesting pod/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 17:15:15.241332 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:15.240962    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf" podUID="e50e437f-d457-49cc-94fc-73aa7340721d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 17:15:15.688984 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:15.688956    4433 scope.go:122] "RemoveContainer" containerID="9289590bd8643ddf4cd8463db57f4a896f7b6d3cbaac8a3645577fb572eff814"
Oct 30 17:15:15.689210 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:15.689190    4433 scope.go:122] "RemoveContainer" containerID="ef613ed0a39d0d01b4897644a221158f521ffaa294d28db74c9df65ba02102c1"
Oct 30 17:15:16.184638 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:16.184548    4433 generic.go:356] "Generic (PLEG): container finished" podID="4e355962-aa3b-4c35-8f1e-b98f41716880" containerID="3e22bd7cb34e9f6640d7ab1ff55a4bebc231a3f60aab7db1db0dfc0da5e6cabe" exitCode=60
Oct 30 17:15:16.184789 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:16.184597    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk" event={"ID":"4e355962-aa3b-4c35-8f1e-b98f41716880","Type":"ContainerDied","Data":"3e22bd7cb34e9f6640d7ab1ff55a4bebc231a3f60aab7db1db0dfc0da5e6cabe"}
Oct 30 17:15:16.184789 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:16.184716    4433 scope.go:122] "RemoveContainer" containerID="3b83496103b70f05fbdc8e573c0b02f2818783804c286d7930ae055449e99aae"
Oct 30 17:15:16.185255 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:16.185237    4433 scope.go:122] "RemoveContainer" containerID="3e22bd7cb34e9f6640d7ab1ff55a4bebc231a3f60aab7db1db0dfc0da5e6cabe"
Oct 30 17:15:16.185728 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:15:16.185638    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk_odh-ai-gateway-infra(4e355962-aa3b-4c35-8f1e-b98f41716880)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk" podUID="4e355962-aa3b-4c35-8f1e-b98f41716880"
Oct 30 17:15:16.215566 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:16.215542    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk"]
Oct 30 17:15:16.240930 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:16.240843    4433 patch_prober.go:29] interesting pod/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 17:15:16.240930 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:16.240906    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf" podUID="e50e437f-d457-49cc-94fc-73aa7340721d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 17:15:16.688180 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:16.688143    4433 scope.go:122] "RemoveContainer" containerID="49fa54dfb5dfbc5cf7b581760369f4472869899728aa1930bfa7129f94b25430"
Oct 30 17:15:16.689197 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:16.688846    4433 scope.go:122] "RemoveContainer" containerID="2cc1d1e387947e3a63aecbabb3d927eb9a0da636f0c877ca5b744c4c3af7d9e0"
Oct 30 17:15:16.689197 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:16.689079    4433 scope.go:122] "RemoveContainer" containerID="bce633a94fe5755bc049521711cf3387b73a487d56e45992f8ff90fe5a678c3e"
Oct 30 17:15:17.203196 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:17.203166    4433 generic.go:356] "Generic (PLEG): container finished" podID="de736fdc-db4a-4b35-b623-5a037e0579d6" containerID="06c2550b2c74452442ebb52e12509c65bf2327ce97e75f1a23b1f9e144962997" exitCode=60
Oct 30 17:15:17.203347 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:17.203310    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889675-4tt58" event={"ID":"de736fdc-db4a-4b35-b623-5a037e0579d6","Type":"ContainerDied","Data":"06c2550b2c74452442ebb52e12509c65bf2327ce97e75f1a23b1f9e144962997"}
Oct 30 17:15:17.203460 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:17.203360    4433 scope.go:122] "RemoveContainer" containerID="9289590bd8643ddf4cd8463db57f4a896f7b6d3cbaac8a3645577fb572eff814"
Oct 30 17:15:17.203820 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:17.203803    4433 scope.go:122] "RemoveContainer" containerID="06c2550b2c74452442ebb52e12509c65bf2327ce97e75f1a23b1f9e144962997"
Oct 30 17:15:17.204295 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:15:17.204268    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-096d75-29889675-4tt58_odh-ai-gateway-infra(de736fdc-db4a-4b35-b623-5a037e0579d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889675-4tt58" podUID="de736fdc-db4a-4b35-b623-5a037e0579d6"
Oct 30 17:15:17.206756 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:17.206736    4433 generic.go:356] "Generic (PLEG): container finished" podID="b0ab47a4-a873-43e7-b528-bc733d58e7a7" containerID="4736915dbd6a448c82dceeb50c018a2a29d476208d31210e8d5cc14c6e4dd5eb" exitCode=60
Oct 30 17:15:17.206844 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:17.206804    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb" event={"ID":"b0ab47a4-a873-43e7-b528-bc733d58e7a7","Type":"ContainerDied","Data":"4736915dbd6a448c82dceeb50c018a2a29d476208d31210e8d5cc14c6e4dd5eb"}
Oct 30 17:15:17.207233 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:17.207216    4433 scope.go:122] "RemoveContainer" containerID="4736915dbd6a448c82dceeb50c018a2a29d476208d31210e8d5cc14c6e4dd5eb"
Oct 30 17:15:17.207589 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:15:17.207569    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb_odh-ai-gateway-infra(b0ab47a4-a873-43e7-b528-bc733d58e7a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb" podUID="b0ab47a4-a873-43e7-b528-bc733d58e7a7"
Oct 30 17:15:17.211477 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:17.211453    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl" event={"ID":"b78c3100-b1b6-4204-af12-54870e4b8dbb","Type":"ContainerStarted","Data":"9dfea3ece8858b2caeb666632b5e5b581d872a2ec5b5d62bfac2bb8f00c60f79"}
Oct 30 17:15:17.219189 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:17.218854    4433 scope.go:122] "RemoveContainer" containerID="ef613ed0a39d0d01b4897644a221158f521ffaa294d28db74c9df65ba02102c1"
Oct 30 17:15:17.241415 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:17.241283    4433 patch_prober.go:29] interesting pod/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 17:15:17.241415 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:17.241347    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf" podUID="e50e437f-d457-49cc-94fc-73aa7340721d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 17:15:17.442056 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:17.442026    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk"
Oct 30 17:15:17.626304 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:17.626208    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e355962-aa3b-4c35-8f1e-b98f41716880-kube-api-access-kv6rh\" (UniqueName: \"kubernetes.io/projected/4e355962-aa3b-4c35-8f1e-b98f41716880-kube-api-access-kv6rh\") pod \"4e355962-aa3b-4c35-8f1e-b98f41716880\" (UID: \"4e355962-aa3b-4c35-8f1e-b98f41716880\") "
Oct 30 17:15:17.626533 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:17.626336    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e355962-aa3b-4c35-8f1e-b98f41716880-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e355962-aa3b-4c35-8f1e-b98f41716880-maas-api-service-ca\") pod \"4e355962-aa3b-4c35-8f1e-b98f41716880\" (UID: \"4e355962-aa3b-4c35-8f1e-b98f41716880\") "
Oct 30 17:15:17.627074 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:17.627038    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e355962-aa3b-4c35-8f1e-b98f41716880-maas-api-service-ca" pod "4e355962-aa3b-4c35-8f1e-b98f41716880" (UID: "4e355962-aa3b-4c35-8f1e-b98f41716880"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:15:17.627387 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:17.627361    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e355962-aa3b-4c35-8f1e-b98f41716880-maas-api-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:15:17.629379 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:17.629348    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e355962-aa3b-4c35-8f1e-b98f41716880-kube-api-access-kv6rh" pod "4e355962-aa3b-4c35-8f1e-b98f41716880" (UID: "4e355962-aa3b-4c35-8f1e-b98f41716880"). InnerVolumeSpecName "kube-api-access-kv6rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:15:17.688942 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:17.688907    4433 scope.go:122] "RemoveContainer" containerID="cb484a471e96e34569aac4184f2dab1b26bdbdadf920e978ea79933d5bd2d078"
Oct 30 17:15:17.690201 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:17.690020    4433 scope.go:122] "RemoveContainer" containerID="dcf9acaddf523341ad6cb7206beba6d1e9f4f1c10dfe5c952d3459882ed9edd3"
Oct 30 17:15:17.729984 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:17.729474    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv6rh\" (UniqueName: \"kubernetes.io/projected/4e355962-aa3b-4c35-8f1e-b98f41716880-kube-api-access-kv6rh\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:15:18.233245 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:18.232954    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk"
Oct 30 17:15:18.233245 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:18.232964    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk" event={"ID":"4e355962-aa3b-4c35-8f1e-b98f41716880","Type":"ContainerDied","Data":"e2a0ecfed1fe9ca1c31d855b45d486c70067c37f0369a21e454c815a1925766a"}
Oct 30 17:15:18.233245 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:18.233018    4433 scope.go:122] "RemoveContainer" containerID="3e22bd7cb34e9f6640d7ab1ff55a4bebc231a3f60aab7db1db0dfc0da5e6cabe"
Oct 30 17:15:18.246433 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:18.245210    4433 patch_prober.go:29] interesting pod/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 17:15:18.246433 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:18.245259    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf" podUID="e50e437f-d457-49cc-94fc-73aa7340721d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 17:15:18.246433 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:18.245793    4433 generic.go:356] "Generic (PLEG): container finished" podID="1aa1ca72-36df-4d56-b3f2-e7b33f4e1379" containerID="ad27f5a4e952722ef6a2a22cbe1add17e067fcb7c8484700b29f33191978f5c5" exitCode=60
Oct 30 17:15:18.246433 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:18.245883    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc" event={"ID":"1aa1ca72-36df-4d56-b3f2-e7b33f4e1379","Type":"ContainerDied","Data":"ad27f5a4e952722ef6a2a22cbe1add17e067fcb7c8484700b29f33191978f5c5"}
Oct 30 17:15:18.246433 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:18.246273    4433 scope.go:122] "RemoveContainer" containerID="ad27f5a4e952722ef6a2a22cbe1add17e067fcb7c8484700b29f33191978f5c5"
Oct 30 17:15:18.247133 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:15:18.246628    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc_odh-ai-gateway-infra(1aa1ca72-36df-4d56-b3f2-e7b33f4e1379)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc" podUID="1aa1ca72-36df-4d56-b3f2-e7b33f4e1379"
Oct 30 17:15:18.249055 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:18.249031    4433 generic.go:356] "Generic (PLEG): container finished" podID="c34089da-9f1d-461a-a3ff-65b5d894eef4" containerID="643d37ac26d2280f4c531b454c10b35958cf1384347c77dc9225516221c651af" exitCode=60
Oct 30 17:15:18.249197 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:18.249131    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889675-djsxq" event={"ID":"c34089da-9f1d-461a-a3ff-65b5d894eef4","Type":"ContainerDied","Data":"643d37ac26d2280f4c531b454c10b35958cf1384347c77dc9225516221c651af"}
Oct 30 17:15:18.249673 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:18.249620    4433 scope.go:122] "RemoveContainer" containerID="2cc1d1e387947e3a63aecbabb3d927eb9a0da636f0c877ca5b744c4c3af7d9e0"
Oct 30 17:15:18.249861 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:18.249845    4433 scope.go:122] "RemoveContainer" containerID="643d37ac26d2280f4c531b454c10b35958cf1384347c77dc9225516221c651af"
Oct 30 17:15:18.250246 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:15:18.250213    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889675-djsxq_odh-ai-gateway-infra(c34089da-9f1d-461a-a3ff-65b5d894eef4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889675-djsxq" podUID="c34089da-9f1d-461a-a3ff-65b5d894eef4"
Oct 30 17:15:18.272255 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:18.272137    4433 generic.go:356] "Generic (PLEG): container finished" podID="b78c3100-b1b6-4204-af12-54870e4b8dbb" containerID="9dfea3ece8858b2caeb666632b5e5b581d872a2ec5b5d62bfac2bb8f00c60f79" exitCode=60
Oct 30 17:15:18.272529 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:18.272458    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl" event={"ID":"b78c3100-b1b6-4204-af12-54870e4b8dbb","Type":"ContainerDied","Data":"9dfea3ece8858b2caeb666632b5e5b581d872a2ec5b5d62bfac2bb8f00c60f79"}
Oct 30 17:15:18.278448 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:18.273255    4433 scope.go:122] "RemoveContainer" containerID="9dfea3ece8858b2caeb666632b5e5b581d872a2ec5b5d62bfac2bb8f00c60f79"
Oct 30 17:15:18.279391 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:15:18.279290    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl_odh-ai-gateway-infra(b78c3100-b1b6-4204-af12-54870e4b8dbb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl" podUID="b78c3100-b1b6-4204-af12-54870e4b8dbb"
Oct 30 17:15:18.297659 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:18.295991    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk"]
Oct 30 17:15:18.307114 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:18.307087    4433 scope.go:122] "RemoveContainer" containerID="49fa54dfb5dfbc5cf7b581760369f4472869899728aa1930bfa7129f94b25430"
Oct 30 17:15:18.307959 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:18.307930    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04248e-29889675-gmdwk"]
Oct 30 17:15:18.324820 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:18.323719    4433 scope.go:122] "RemoveContainer" containerID="bce633a94fe5755bc049521711cf3387b73a487d56e45992f8ff90fe5a678c3e"
Oct 30 17:15:18.336080 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:18.336032    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb"]
Oct 30 17:15:18.390077 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:18.390036    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl"]
Oct 30 17:15:18.487793 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:18.487690    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889675-4tt58"]
Oct 30 17:15:18.581529 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:18.581472    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb"
Oct 30 17:15:18.694294 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:18.694257    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0ab47a4-a873-43e7-b528-bc733d58e7a7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ab47a4-a873-43e7-b528-bc733d58e7a7-maas-api-service-ca\") pod \"b0ab47a4-a873-43e7-b528-bc733d58e7a7\" (UID: \"b0ab47a4-a873-43e7-b528-bc733d58e7a7\") "
Oct 30 17:15:18.694707 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:18.694655    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0ab47a4-a873-43e7-b528-bc733d58e7a7-kube-api-access-tqxtr\" (UniqueName: \"kubernetes.io/projected/b0ab47a4-a873-43e7-b528-bc733d58e7a7-kube-api-access-tqxtr\") pod \"b0ab47a4-a873-43e7-b528-bc733d58e7a7\" (UID: \"b0ab47a4-a873-43e7-b528-bc733d58e7a7\") "
Oct 30 17:15:18.696517 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:18.695615    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0ab47a4-a873-43e7-b528-bc733d58e7a7-maas-api-service-ca" pod "b0ab47a4-a873-43e7-b528-bc733d58e7a7" (UID: "b0ab47a4-a873-43e7-b528-bc733d58e7a7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:15:18.699588 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:18.699464    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ab47a4-a873-43e7-b528-bc733d58e7a7-kube-api-access-tqxtr" pod "b0ab47a4-a873-43e7-b528-bc733d58e7a7" (UID: "b0ab47a4-a873-43e7-b528-bc733d58e7a7"). InnerVolumeSpecName "kube-api-access-tqxtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:15:18.796942 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:18.796915    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqxtr\" (UniqueName: \"kubernetes.io/projected/b0ab47a4-a873-43e7-b528-bc733d58e7a7-kube-api-access-tqxtr\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:15:18.796942 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:18.796941    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ab47a4-a873-43e7-b528-bc733d58e7a7-maas-api-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:15:18.814954 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:18.814930    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889675-4tt58"
Oct 30 17:15:18.898368 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:18.898320    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de736fdc-db4a-4b35-b623-5a037e0579d6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de736fdc-db4a-4b35-b623-5a037e0579d6-maas-api-service-ca\") pod \"de736fdc-db4a-4b35-b623-5a037e0579d6\" (UID: \"de736fdc-db4a-4b35-b623-5a037e0579d6\") "
Oct 30 17:15:18.898611 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:18.898574    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de736fdc-db4a-4b35-b623-5a037e0579d6-kube-api-access-k5jmx\" (UniqueName: \"kubernetes.io/projected/de736fdc-db4a-4b35-b623-5a037e0579d6-kube-api-access-k5jmx\") pod \"de736fdc-db4a-4b35-b623-5a037e0579d6\" (UID: \"de736fdc-db4a-4b35-b623-5a037e0579d6\") "
Oct 30 17:15:18.899746 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:18.899712    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de736fdc-db4a-4b35-b623-5a037e0579d6-maas-api-service-ca" pod "de736fdc-db4a-4b35-b623-5a037e0579d6" (UID: "de736fdc-db4a-4b35-b623-5a037e0579d6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:15:18.901734 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:18.901706    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de736fdc-db4a-4b35-b623-5a037e0579d6-kube-api-access-k5jmx" pod "de736fdc-db4a-4b35-b623-5a037e0579d6" (UID: "de736fdc-db4a-4b35-b623-5a037e0579d6"). InnerVolumeSpecName "kube-api-access-k5jmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:15:19.001348 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.001253    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5jmx\" (UniqueName: \"kubernetes.io/projected/de736fdc-db4a-4b35-b623-5a037e0579d6-kube-api-access-k5jmx\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:15:19.001348 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.001289    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de736fdc-db4a-4b35-b623-5a037e0579d6-maas-api-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:15:19.240597 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.240549    4433 patch_prober.go:29] interesting pod/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 17:15:19.240796 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.240619    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf" podUID="e50e437f-d457-49cc-94fc-73aa7340721d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 17:15:19.282392 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.282302    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889675-4tt58"
Oct 30 17:15:19.282392 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.282331    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889675-4tt58" event={"ID":"de736fdc-db4a-4b35-b623-5a037e0579d6","Type":"ContainerDied","Data":"87ee5a295800561fbd1b75ea5f4052678d6cf36af094bca65ec96205a7406594"}
Oct 30 17:15:19.282931 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.282424    4433 scope.go:122] "RemoveContainer" containerID="06c2550b2c74452442ebb52e12509c65bf2327ce97e75f1a23b1f9e144962997"
Oct 30 17:15:19.283935 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.283903    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb"
Oct 30 17:15:19.283935 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.283914    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb" event={"ID":"b0ab47a4-a873-43e7-b528-bc733d58e7a7","Type":"ContainerDied","Data":"cf8dc49948d4b439acda329e86a4c8a3f34943895f8a0020bb3792b8ebded437"}
Oct 30 17:15:19.292993 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.292964    4433 generic.go:356] "Generic (PLEG): container finished" podID="ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b" containerID="b5e1383f0d17af8334c5aa1fbb12339e252706f376fdc6508b4ea6b7e34a6b28" exitCode=60
Oct 30 17:15:19.293146 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.293039    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr" event={"ID":"ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b","Type":"ContainerDied","Data":"b5e1383f0d17af8334c5aa1fbb12339e252706f376fdc6508b4ea6b7e34a6b28"}
Oct 30 17:15:19.293564 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.293543    4433 scope.go:122] "RemoveContainer" containerID="b5e1383f0d17af8334c5aa1fbb12339e252706f376fdc6508b4ea6b7e34a6b28"
Oct 30 17:15:19.294083 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:15:19.293918    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr_odh-ai-gateway-infra(ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr" podUID="ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b"
Oct 30 17:15:19.296340 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.296300    4433 scope.go:122] "RemoveContainer" containerID="4736915dbd6a448c82dceeb50c018a2a29d476208d31210e8d5cc14c6e4dd5eb"
Oct 30 17:15:19.298480 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.298454    4433 generic.go:356] "Generic (PLEG): container finished" podID="938091c7-0d73-4944-a33b-f123c256a2a4" containerID="a09dff90f712a8cbaaf8c90e91ba80dc7a04ada5ba5626f83c67f95d99b3cd0d" exitCode=60
Oct 30 17:15:19.298557 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.298524    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl" event={"ID":"938091c7-0d73-4944-a33b-f123c256a2a4","Type":"ContainerDied","Data":"a09dff90f712a8cbaaf8c90e91ba80dc7a04ada5ba5626f83c67f95d99b3cd0d"}
Oct 30 17:15:19.299062 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.299038    4433 scope.go:122] "RemoveContainer" containerID="a09dff90f712a8cbaaf8c90e91ba80dc7a04ada5ba5626f83c67f95d99b3cd0d"
Oct 30 17:15:19.299482 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:15:19.299454    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl_odh-ai-gateway-infra(938091c7-0d73-4944-a33b-f123c256a2a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl" podUID="938091c7-0d73-4944-a33b-f123c256a2a4"
Oct 30 17:15:19.307757 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.307735    4433 scope.go:122] "RemoveContainer" containerID="cb484a471e96e34569aac4184f2dab1b26bdbdadf920e978ea79933d5bd2d078"
Oct 30 17:15:19.333245 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.326577    4433 scope.go:122] "RemoveContainer" containerID="dcf9acaddf523341ad6cb7206beba6d1e9f4f1c10dfe5c952d3459882ed9edd3"
Oct 30 17:15:19.355188 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.355136    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc"]
Oct 30 17:15:19.388440 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.380823    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb"]
Oct 30 17:15:19.401438 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.396816    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b80d83-29889675-xx5hb"]
Oct 30 17:15:19.465566 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.459966    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889675-djsxq"]
Oct 30 17:15:19.543462 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.543360    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889675-4tt58"]
Oct 30 17:15:19.545288 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.545259    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-096d75-29889675-4tt58"]
Oct 30 17:15:19.561399 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.561185    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl"
Oct 30 17:15:19.621786 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.621753    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc"
Oct 30 17:15:19.630772 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.630741    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aa1ca72-36df-4d56-b3f2-e7b33f4e1379-kube-api-access-rncx2\" (UniqueName: \"kubernetes.io/projected/1aa1ca72-36df-4d56-b3f2-e7b33f4e1379-kube-api-access-rncx2\") pod \"1aa1ca72-36df-4d56-b3f2-e7b33f4e1379\" (UID: \"1aa1ca72-36df-4d56-b3f2-e7b33f4e1379\") "
Oct 30 17:15:19.630935 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.630796    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b78c3100-b1b6-4204-af12-54870e4b8dbb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b78c3100-b1b6-4204-af12-54870e4b8dbb-maas-api-service-ca\") pod \"b78c3100-b1b6-4204-af12-54870e4b8dbb\" (UID: \"b78c3100-b1b6-4204-af12-54870e4b8dbb\") "
Oct 30 17:15:19.630935 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.630832    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aa1ca72-36df-4d56-b3f2-e7b33f4e1379-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa1ca72-36df-4d56-b3f2-e7b33f4e1379-maas-api-service-ca\") pod \"1aa1ca72-36df-4d56-b3f2-e7b33f4e1379\" (UID: \"1aa1ca72-36df-4d56-b3f2-e7b33f4e1379\") "
Oct 30 17:15:19.630935 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.630912    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b78c3100-b1b6-4204-af12-54870e4b8dbb-kube-api-access-tw8vs\" (UniqueName: \"kubernetes.io/projected/b78c3100-b1b6-4204-af12-54870e4b8dbb-kube-api-access-tw8vs\") pod \"b78c3100-b1b6-4204-af12-54870e4b8dbb\" (UID: \"b78c3100-b1b6-4204-af12-54870e4b8dbb\") "
Oct 30 17:15:19.631392 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.631362    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aa1ca72-36df-4d56-b3f2-e7b33f4e1379-maas-api-service-ca" pod "1aa1ca72-36df-4d56-b3f2-e7b33f4e1379" (UID: "1aa1ca72-36df-4d56-b3f2-e7b33f4e1379"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:15:19.631660 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.631638    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b78c3100-b1b6-4204-af12-54870e4b8dbb-maas-api-service-ca" pod "b78c3100-b1b6-4204-af12-54870e4b8dbb" (UID: "b78c3100-b1b6-4204-af12-54870e4b8dbb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:15:19.634968 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.634941    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b78c3100-b1b6-4204-af12-54870e4b8dbb-kube-api-access-tw8vs" pod "b78c3100-b1b6-4204-af12-54870e4b8dbb" (UID: "b78c3100-b1b6-4204-af12-54870e4b8dbb"). InnerVolumeSpecName "kube-api-access-tw8vs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:15:19.639159 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.639042    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aa1ca72-36df-4d56-b3f2-e7b33f4e1379-kube-api-access-rncx2" pod "1aa1ca72-36df-4d56-b3f2-e7b33f4e1379" (UID: "1aa1ca72-36df-4d56-b3f2-e7b33f4e1379"). InnerVolumeSpecName "kube-api-access-rncx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:15:19.702460 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.702424    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e355962-aa3b-4c35-8f1e-b98f41716880" path="/var/lib/kubelet/pods/4e355962-aa3b-4c35-8f1e-b98f41716880/volumes"
Oct 30 17:15:19.703468 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.703444    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0ab47a4-a873-43e7-b528-bc733d58e7a7" path="/var/lib/kubelet/pods/b0ab47a4-a873-43e7-b528-bc733d58e7a7/volumes"
Oct 30 17:15:19.704272 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.704251    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de736fdc-db4a-4b35-b623-5a037e0579d6" path="/var/lib/kubelet/pods/de736fdc-db4a-4b35-b623-5a037e0579d6/volumes"
Oct 30 17:15:19.731365 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.731331    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889675-djsxq"
Oct 30 17:15:19.733024 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.732995    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rncx2\" (UniqueName: \"kubernetes.io/projected/1aa1ca72-36df-4d56-b3f2-e7b33f4e1379-kube-api-access-rncx2\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:15:19.733202 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.733185    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b78c3100-b1b6-4204-af12-54870e4b8dbb-maas-api-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:15:19.733303 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.733290    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa1ca72-36df-4d56-b3f2-e7b33f4e1379-maas-api-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:15:19.733546 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.733518    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw8vs\" (UniqueName: \"kubernetes.io/projected/b78c3100-b1b6-4204-af12-54870e4b8dbb-kube-api-access-tw8vs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:15:19.835247 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.835161    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c34089da-9f1d-461a-a3ff-65b5d894eef4-kube-api-access-pzwqn\" (UniqueName: \"kubernetes.io/projected/c34089da-9f1d-461a-a3ff-65b5d894eef4-kube-api-access-pzwqn\") pod \"c34089da-9f1d-461a-a3ff-65b5d894eef4\" (UID: \"c34089da-9f1d-461a-a3ff-65b5d894eef4\") "
Oct 30 17:15:19.835438 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.835278    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c34089da-9f1d-461a-a3ff-65b5d894eef4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c34089da-9f1d-461a-a3ff-65b5d894eef4-maas-api-service-ca\") pod \"c34089da-9f1d-461a-a3ff-65b5d894eef4\" (UID: \"c34089da-9f1d-461a-a3ff-65b5d894eef4\") "
Oct 30 17:15:19.835699 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.835674    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c34089da-9f1d-461a-a3ff-65b5d894eef4-maas-api-service-ca" pod "c34089da-9f1d-461a-a3ff-65b5d894eef4" (UID: "c34089da-9f1d-461a-a3ff-65b5d894eef4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:15:19.835936 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.835918    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c34089da-9f1d-461a-a3ff-65b5d894eef4-maas-api-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:15:19.837291 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.837262    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c34089da-9f1d-461a-a3ff-65b5d894eef4-kube-api-access-pzwqn" pod "c34089da-9f1d-461a-a3ff-65b5d894eef4" (UID: "c34089da-9f1d-461a-a3ff-65b5d894eef4"). InnerVolumeSpecName "kube-api-access-pzwqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:15:19.937140 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:19.937102    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzwqn\" (UniqueName: \"kubernetes.io/projected/c34089da-9f1d-461a-a3ff-65b5d894eef4-kube-api-access-pzwqn\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:15:20.245750 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:20.245707    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"
Oct 30 17:15:20.246121 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:20.246098    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"
Oct 30 17:15:20.247189 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:20.247165    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"
Oct 30 17:15:20.308502 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:20.308439    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc"
Oct 30 17:15:20.309004 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:20.308439    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc" event={"ID":"1aa1ca72-36df-4d56-b3f2-e7b33f4e1379","Type":"ContainerDied","Data":"a88ab0e9f227d385caf33d9707c49fc749aa5cffc80bfb5ceb98714d45b51e32"}
Oct 30 17:15:20.309004 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:20.308591    4433 scope.go:122] "RemoveContainer" containerID="ad27f5a4e952722ef6a2a22cbe1add17e067fcb7c8484700b29f33191978f5c5"
Oct 30 17:15:20.310170 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:20.310143    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889675-djsxq"
Oct 30 17:15:20.310316 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:20.310291    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889675-djsxq" event={"ID":"c34089da-9f1d-461a-a3ff-65b5d894eef4","Type":"ContainerDied","Data":"381456db0737707a95aeeb6cfbb557297b441219cb1fd5bfcd0763174d5691b8"}
Oct 30 17:15:20.314197 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:20.314178    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl"
Oct 30 17:15:20.314635 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:20.314614    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl" event={"ID":"b78c3100-b1b6-4204-af12-54870e4b8dbb","Type":"ContainerDied","Data":"31f4b375390f6c8b3c3881890c4e3675e89a9d6794f0dc50f605d154b1ab444a"}
Oct 30 17:15:20.321734 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:20.321713    4433 scope.go:122] "RemoveContainer" containerID="643d37ac26d2280f4c531b454c10b35958cf1384347c77dc9225516221c651af"
Oct 30 17:15:20.330948 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:20.330931    4433 scope.go:122] "RemoveContainer" containerID="9dfea3ece8858b2caeb666632b5e5b581d872a2ec5b5d62bfac2bb8f00c60f79"
Oct 30 17:15:20.361886 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:20.361843    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr"]
Oct 30 17:15:20.392575 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:20.392498    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl"]
Oct 30 17:15:20.397169 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:20.397137    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d003a5-29889675-t7hdl"]
Oct 30 17:15:20.433319 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:20.432535    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889675-djsxq"]
Oct 30 17:15:20.440464 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:20.440064    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889675-djsxq"]
Oct 30 17:15:20.478020 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:20.474707    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc"]
Oct 30 17:15:20.484139 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:20.484093    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889675-lcbcc"]
Oct 30 17:15:20.606356 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:20.606328    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr"
Oct 30 17:15:20.648601 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:20.648552    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b-maas-api-service-ca\") pod \"ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b\" (UID: \"ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b\") "
Oct 30 17:15:20.648805 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:20.648670    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b-kube-api-access-cjkqh\" (UniqueName: \"kubernetes.io/projected/ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b-kube-api-access-cjkqh\") pod \"ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b\" (UID: \"ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b\") "
Oct 30 17:15:20.649077 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:20.649010    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b-maas-api-service-ca" pod "ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b" (UID: "ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:15:20.649548 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:20.649526    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b-maas-api-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:15:20.651278 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:20.651252    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b-kube-api-access-cjkqh" pod "ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b" (UID: "ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b"). InnerVolumeSpecName "kube-api-access-cjkqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:15:20.659177 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:20.659146    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl"]
Oct 30 17:15:20.752030 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:20.751996    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjkqh\" (UniqueName: \"kubernetes.io/projected/ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b-kube-api-access-cjkqh\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:15:20.854530 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:20.854498    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl"
Oct 30 17:15:20.954897 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:20.954839    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/938091c7-0d73-4944-a33b-f123c256a2a4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/938091c7-0d73-4944-a33b-f123c256a2a4-maas-api-service-ca\") pod \"938091c7-0d73-4944-a33b-f123c256a2a4\" (UID: \"938091c7-0d73-4944-a33b-f123c256a2a4\") "
Oct 30 17:15:20.954897 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:20.954904    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/938091c7-0d73-4944-a33b-f123c256a2a4-kube-api-access-xk5mr\" (UniqueName: \"kubernetes.io/projected/938091c7-0d73-4944-a33b-f123c256a2a4-kube-api-access-xk5mr\") pod \"938091c7-0d73-4944-a33b-f123c256a2a4\" (UID: \"938091c7-0d73-4944-a33b-f123c256a2a4\") "
Oct 30 17:15:20.955711 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:20.955676    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/938091c7-0d73-4944-a33b-f123c256a2a4-maas-api-service-ca" pod "938091c7-0d73-4944-a33b-f123c256a2a4" (UID: "938091c7-0d73-4944-a33b-f123c256a2a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:15:20.956088 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:20.956062    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/938091c7-0d73-4944-a33b-f123c256a2a4-maas-api-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:15:20.957540 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:20.957518    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/938091c7-0d73-4944-a33b-f123c256a2a4-kube-api-access-xk5mr" pod "938091c7-0d73-4944-a33b-f123c256a2a4" (UID: "938091c7-0d73-4944-a33b-f123c256a2a4"). InnerVolumeSpecName "kube-api-access-xk5mr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:15:21.057924 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:21.057884    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk5mr\" (UniqueName: \"kubernetes.io/projected/938091c7-0d73-4944-a33b-f123c256a2a4-kube-api-access-xk5mr\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:15:21.332699 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:21.332667    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr"
Oct 30 17:15:21.333177 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:21.332670    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr" event={"ID":"ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b","Type":"ContainerDied","Data":"ab8f4b94f8b3854bf4d1510bb8ed73cd4443dfd1e81e09596d8bc0a2569c5ed1"}
Oct 30 17:15:21.333177 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:21.332819    4433 scope.go:122] "RemoveContainer" containerID="b5e1383f0d17af8334c5aa1fbb12339e252706f376fdc6508b4ea6b7e34a6b28"
Oct 30 17:15:21.337784 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:21.337746    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl"
Oct 30 17:15:21.337915 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:21.337855    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl" event={"ID":"938091c7-0d73-4944-a33b-f123c256a2a4","Type":"ContainerDied","Data":"1dc7a5b76e95a120eae2f740d61fb76bb525818188cb09c7965c978c2ccd1867"}
Oct 30 17:15:21.347000 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:21.346881    4433 scope.go:122] "RemoveContainer" containerID="a09dff90f712a8cbaaf8c90e91ba80dc7a04ada5ba5626f83c67f95d99b3cd0d"
Oct 30 17:15:21.389503 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:21.389466    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr"]
Oct 30 17:15:21.415615 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:21.415583    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d384dc-29889675-5r2hr"]
Oct 30 17:15:21.455751 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:21.455698    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl"]
Oct 30 17:15:21.464029 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:21.463990    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889675-t5zhl"]
Oct 30 17:15:21.696696 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:21.696660    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aa1ca72-36df-4d56-b3f2-e7b33f4e1379" path="/var/lib/kubelet/pods/1aa1ca72-36df-4d56-b3f2-e7b33f4e1379/volumes"
Oct 30 17:15:21.697192 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:21.697177    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="938091c7-0d73-4944-a33b-f123c256a2a4" path="/var/lib/kubelet/pods/938091c7-0d73-4944-a33b-f123c256a2a4/volumes"
Oct 30 17:15:21.697617 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:21.697603    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b78c3100-b1b6-4204-af12-54870e4b8dbb" path="/var/lib/kubelet/pods/b78c3100-b1b6-4204-af12-54870e4b8dbb/volumes"
Oct 30 17:15:21.698296 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:21.698279    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c34089da-9f1d-461a-a3ff-65b5d894eef4" path="/var/lib/kubelet/pods/c34089da-9f1d-461a-a3ff-65b5d894eef4/volumes"
Oct 30 17:15:21.698716 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:21.698701    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b" path="/var/lib/kubelet/pods/ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b/volumes"
Oct 30 17:15:22.499318 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.499276    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb"]
Oct 30 17:15:22.500269 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500243    4433 cpu_manager.go:414] "containerMap: removing container" podUID="b0ab47a4-a873-43e7-b528-bc733d58e7a7" containerName="cleanup"
Oct 30 17:15:22.500318 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500273    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ab47a4-a873-43e7-b528-bc733d58e7a7" containerName="cleanup"
Oct 30 17:15:22.500318 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500293    4433 cpu_manager.go:414] "containerMap: removing container" podUID="938091c7-0d73-4944-a33b-f123c256a2a4" containerName="cleanup"
Oct 30 17:15:22.500318 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500301    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="938091c7-0d73-4944-a33b-f123c256a2a4" containerName="cleanup"
Oct 30 17:15:22.500318 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500312    4433 cpu_manager.go:414] "containerMap: removing container" podUID="ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b" containerName="cleanup"
Oct 30 17:15:22.500473 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500320    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b" containerName="cleanup"
Oct 30 17:15:22.500473 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500329    4433 cpu_manager.go:414] "containerMap: removing container" podUID="938091c7-0d73-4944-a33b-f123c256a2a4" containerName="cleanup"
Oct 30 17:15:22.500473 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500337    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="938091c7-0d73-4944-a33b-f123c256a2a4" containerName="cleanup"
Oct 30 17:15:22.500473 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500353    4433 cpu_manager.go:414] "containerMap: removing container" podUID="de736fdc-db4a-4b35-b623-5a037e0579d6" containerName="cleanup"
Oct 30 17:15:22.500473 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500360    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de736fdc-db4a-4b35-b623-5a037e0579d6" containerName="cleanup"
Oct 30 17:15:22.500473 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500375    4433 cpu_manager.go:414] "containerMap: removing container" podUID="ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b" containerName="cleanup"
Oct 30 17:15:22.500473 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500382    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b" containerName="cleanup"
Oct 30 17:15:22.500473 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500393    4433 cpu_manager.go:414] "containerMap: removing container" podUID="1aa1ca72-36df-4d56-b3f2-e7b33f4e1379" containerName="cleanup"
Oct 30 17:15:22.500473 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500416    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aa1ca72-36df-4d56-b3f2-e7b33f4e1379" containerName="cleanup"
Oct 30 17:15:22.500473 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500428    4433 cpu_manager.go:414] "containerMap: removing container" podUID="b78c3100-b1b6-4204-af12-54870e4b8dbb" containerName="cleanup"
Oct 30 17:15:22.500473 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500434    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b78c3100-b1b6-4204-af12-54870e4b8dbb" containerName="cleanup"
Oct 30 17:15:22.500473 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500447    4433 cpu_manager.go:414] "containerMap: removing container" podUID="b78c3100-b1b6-4204-af12-54870e4b8dbb" containerName="cleanup"
Oct 30 17:15:22.500473 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500454    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b78c3100-b1b6-4204-af12-54870e4b8dbb" containerName="cleanup"
Oct 30 17:15:22.500473 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500469    4433 cpu_manager.go:414] "containerMap: removing container" podUID="4e355962-aa3b-4c35-8f1e-b98f41716880" containerName="cleanup"
Oct 30 17:15:22.500473 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500476    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e355962-aa3b-4c35-8f1e-b98f41716880" containerName="cleanup"
Oct 30 17:15:22.500897 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500491    4433 cpu_manager.go:414] "containerMap: removing container" podUID="b0ab47a4-a873-43e7-b528-bc733d58e7a7" containerName="cleanup"
Oct 30 17:15:22.500897 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500499    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ab47a4-a873-43e7-b528-bc733d58e7a7" containerName="cleanup"
Oct 30 17:15:22.500897 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500512    4433 cpu_manager.go:414] "containerMap: removing container" podUID="4e355962-aa3b-4c35-8f1e-b98f41716880" containerName="cleanup"
Oct 30 17:15:22.500897 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500520    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e355962-aa3b-4c35-8f1e-b98f41716880" containerName="cleanup"
Oct 30 17:15:22.500897 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500535    4433 cpu_manager.go:414] "containerMap: removing container" podUID="c34089da-9f1d-461a-a3ff-65b5d894eef4" containerName="cleanup"
Oct 30 17:15:22.500897 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500541    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c34089da-9f1d-461a-a3ff-65b5d894eef4" containerName="cleanup"
Oct 30 17:15:22.500897 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500552    4433 cpu_manager.go:414] "containerMap: removing container" podUID="ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b" containerName="cleanup"
Oct 30 17:15:22.500897 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500559    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b" containerName="cleanup"
Oct 30 17:15:22.500897 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500578    4433 cpu_manager.go:414] "containerMap: removing container" podUID="1aa1ca72-36df-4d56-b3f2-e7b33f4e1379" containerName="cleanup"
Oct 30 17:15:22.500897 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500585    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aa1ca72-36df-4d56-b3f2-e7b33f4e1379" containerName="cleanup"
Oct 30 17:15:22.500897 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500604    4433 cpu_manager.go:414] "containerMap: removing container" podUID="c34089da-9f1d-461a-a3ff-65b5d894eef4" containerName="cleanup"
Oct 30 17:15:22.500897 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500611    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c34089da-9f1d-461a-a3ff-65b5d894eef4" containerName="cleanup"
Oct 30 17:15:22.500897 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500624    4433 cpu_manager.go:414] "containerMap: removing container" podUID="1aa1ca72-36df-4d56-b3f2-e7b33f4e1379" containerName="cleanup"
Oct 30 17:15:22.500897 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500630    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aa1ca72-36df-4d56-b3f2-e7b33f4e1379" containerName="cleanup"
Oct 30 17:15:22.500897 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500641    4433 cpu_manager.go:414] "containerMap: removing container" podUID="c34089da-9f1d-461a-a3ff-65b5d894eef4" containerName="cleanup"
Oct 30 17:15:22.500897 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500648    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c34089da-9f1d-461a-a3ff-65b5d894eef4" containerName="cleanup"
Oct 30 17:15:22.500897 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500667    4433 cpu_manager.go:414] "containerMap: removing container" podUID="b78c3100-b1b6-4204-af12-54870e4b8dbb" containerName="cleanup"
Oct 30 17:15:22.500897 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500673    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b78c3100-b1b6-4204-af12-54870e4b8dbb" containerName="cleanup"
Oct 30 17:15:22.500897 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500697    4433 cpu_manager.go:414] "containerMap: removing container" podUID="de736fdc-db4a-4b35-b623-5a037e0579d6" containerName="cleanup"
Oct 30 17:15:22.500897 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500704    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de736fdc-db4a-4b35-b623-5a037e0579d6" containerName="cleanup"
Oct 30 17:15:22.500897 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500852    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aa1ca72-36df-4d56-b3f2-e7b33f4e1379" containerName="cleanup"
Oct 30 17:15:22.500897 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500866    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e355962-aa3b-4c35-8f1e-b98f41716880" containerName="cleanup"
Oct 30 17:15:22.500897 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500878    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="c34089da-9f1d-461a-a3ff-65b5d894eef4" containerName="cleanup"
Oct 30 17:15:22.500897 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500888    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aa1ca72-36df-4d56-b3f2-e7b33f4e1379" containerName="cleanup"
Oct 30 17:15:22.500897 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500895    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="c34089da-9f1d-461a-a3ff-65b5d894eef4" containerName="cleanup"
Oct 30 17:15:22.500897 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500907    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="b78c3100-b1b6-4204-af12-54870e4b8dbb" containerName="cleanup"
Oct 30 17:15:22.501655 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500919    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="de736fdc-db4a-4b35-b623-5a037e0579d6" containerName="cleanup"
Oct 30 17:15:22.501655 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500927    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ab47a4-a873-43e7-b528-bc733d58e7a7" containerName="cleanup"
Oct 30 17:15:22.501655 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500935    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="938091c7-0d73-4944-a33b-f123c256a2a4" containerName="cleanup"
Oct 30 17:15:22.501655 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500943    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b" containerName="cleanup"
Oct 30 17:15:22.501655 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500951    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="938091c7-0d73-4944-a33b-f123c256a2a4" containerName="cleanup"
Oct 30 17:15:22.501655 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500959    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="de736fdc-db4a-4b35-b623-5a037e0579d6" containerName="cleanup"
Oct 30 17:15:22.501655 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500968    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b" containerName="cleanup"
Oct 30 17:15:22.501655 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500978    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="b78c3100-b1b6-4204-af12-54870e4b8dbb" containerName="cleanup"
Oct 30 17:15:22.501655 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500988    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ab47a4-a873-43e7-b528-bc733d58e7a7" containerName="cleanup"
Oct 30 17:15:22.501655 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.500997    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e355962-aa3b-4c35-8f1e-b98f41716880" containerName="cleanup"
Oct 30 17:15:22.501655 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.501004    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="938091c7-0d73-4944-a33b-f123c256a2a4" containerName="cleanup"
Oct 30 17:15:22.506556 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.506534    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb"
Oct 30 17:15:22.522882 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.522844    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb"]
Oct 30 17:15:22.529218 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.529188    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-7c52f8-serving-cert\""
Oct 30 17:15:22.580071 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.580041    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb27ba4a-4385-4f07-a202-f8f3bccb6559-maas-api-tls\") pod \"maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb\" (UID: \"eb27ba4a-4385-4f07-a202-f8f3bccb6559\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb"
Oct 30 17:15:22.580541 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.580186    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb27ba4a-4385-4f07-a202-f8f3bccb6559-metrics-certs\") pod \"maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb\" (UID: \"eb27ba4a-4385-4f07-a202-f8f3bccb6559\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb"
Oct 30 17:15:22.580541 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.580260    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb27ba4a-4385-4f07-a202-f8f3bccb6559-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb\" (UID: \"eb27ba4a-4385-4f07-a202-f8f3bccb6559\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb"
Oct 30 17:15:22.580541 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.580339    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vmdx\" (UniqueName: \"kubernetes.io/projected/eb27ba4a-4385-4f07-a202-f8f3bccb6559-kube-api-access-2vmdx\") pod \"maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb\" (UID: \"eb27ba4a-4385-4f07-a202-f8f3bccb6559\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb"
Oct 30 17:15:22.682094 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.682062    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb27ba4a-4385-4f07-a202-f8f3bccb6559-maas-api-tls\") pod \"maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb\" (UID: \"eb27ba4a-4385-4f07-a202-f8f3bccb6559\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb"
Oct 30 17:15:22.682260 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.682169    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb27ba4a-4385-4f07-a202-f8f3bccb6559-metrics-certs\") pod \"maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb\" (UID: \"eb27ba4a-4385-4f07-a202-f8f3bccb6559\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb"
Oct 30 17:15:22.682260 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.682206    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb27ba4a-4385-4f07-a202-f8f3bccb6559-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb\" (UID: \"eb27ba4a-4385-4f07-a202-f8f3bccb6559\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb"
Oct 30 17:15:22.682260 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.682254    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vmdx\" (UniqueName: \"kubernetes.io/projected/eb27ba4a-4385-4f07-a202-f8f3bccb6559-kube-api-access-2vmdx\") pod \"maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb\" (UID: \"eb27ba4a-4385-4f07-a202-f8f3bccb6559\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb"
Oct 30 17:15:22.682910 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.682880    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb27ba4a-4385-4f07-a202-f8f3bccb6559-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb\" (UID: \"eb27ba4a-4385-4f07-a202-f8f3bccb6559\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb"
Oct 30 17:15:22.684760 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.684730    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb27ba4a-4385-4f07-a202-f8f3bccb6559-maas-api-tls\") pod \"maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb\" (UID: \"eb27ba4a-4385-4f07-a202-f8f3bccb6559\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb"
Oct 30 17:15:22.684866 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.684736    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb27ba4a-4385-4f07-a202-f8f3bccb6559-metrics-certs\") pod \"maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb\" (UID: \"eb27ba4a-4385-4f07-a202-f8f3bccb6559\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb"
Oct 30 17:15:22.704775 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.704751    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vmdx\" (UniqueName: \"kubernetes.io/projected/eb27ba4a-4385-4f07-a202-f8f3bccb6559-kube-api-access-2vmdx\") pod \"maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb\" (UID: \"eb27ba4a-4385-4f07-a202-f8f3bccb6559\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb"
Oct 30 17:15:22.829542 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:22.829458    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb"
Oct 30 17:15:23.137842 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:23.137813    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb"]
Oct 30 17:15:23.139772 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:15:23.139741    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb27ba4a_4385_4f07_a202_f8f3bccb6559.slice/crio-30b5d4ec005e55058dd6edbd3f9daf07ff80c93f9879fca8eab758446205eac5 WatchSource:0}: Error finding container 30b5d4ec005e55058dd6edbd3f9daf07ff80c93f9879fca8eab758446205eac5: Status 404 returned error can't find the container with id 30b5d4ec005e55058dd6edbd3f9daf07ff80c93f9879fca8eab758446205eac5
Oct 30 17:15:23.356307 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:23.356273    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb" event={"ID":"eb27ba4a-4385-4f07-a202-f8f3bccb6559","Type":"ContainerStarted","Data":"30b5d4ec005e55058dd6edbd3f9daf07ff80c93f9879fca8eab758446205eac5"}
Oct 30 17:15:24.364484 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:24.364439    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb" event={"ID":"eb27ba4a-4385-4f07-a202-f8f3bccb6559","Type":"ContainerStarted","Data":"faf2df524894edf5df4fbde63fede5d1594b671ed691324b4e59faaec7c57709"}
Oct 30 17:15:24.364880 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:24.364532    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb"
Oct 30 17:15:24.406634 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:24.406572    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb" podStartSLOduration=2.069118739 podStartE2EDuration="2.40655611s" totalImagesPullingTime="337.437371ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:24.40489102 +0000 UTC m=+1700.839793245" watchObservedRunningTime="2026-10-30 17:15:24.40655611 +0000 UTC m=+1700.841458316"
Oct 30 17:15:25.497566 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:25.497533    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"]
Oct 30 17:15:25.498022 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:25.497826    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s" podUID="16eaa2d1-45c5-4b91-8638-e6a5c492de5c" containerName="istio-proxy" containerID="cri-o://68d3078d10323e47f2fa32f2fbf0cfea638f5553493f1fa8d6de7dbc1b0adb0c" gracePeriod=30
Oct 30 17:15:26.978730 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:15:26.978685    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:15:30.377319 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:30.377287    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb"
Oct 30 17:15:31.184812 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.184786    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"
Oct 30 17:15:31.336841 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.336744    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-kube-api-access-2gfkg\" (UniqueName: \"kubernetes.io/projected/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-kube-api-access-2gfkg\") pod \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") "
Oct 30 17:15:31.337024 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.336848    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-credential-socket\") pod \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") "
Oct 30 17:15:31.337024 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.336902    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-workload-socket\") pod \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") "
Oct 30 17:15:31.337024 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.336932    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-istiod-ca-cert\") pod \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") "
Oct 30 17:15:31.337024 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.336988    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-workload-certs\") pod \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") "
Oct 30 17:15:31.337235 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.337073    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-istio-data\") pod \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") "
Oct 30 17:15:31.337235 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.337147    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-credential-socket" pod "16eaa2d1-45c5-4b91-8638-e6a5c492de5c" (UID: "16eaa2d1-45c5-4b91-8638-e6a5c492de5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:15:31.337235 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.337171    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-istio-podinfo\") pod \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") "
Oct 30 17:15:31.337481 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.337292    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-istio-token\" (UniqueName: \"kubernetes.io/projected/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-istio-token\") pod \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") "
Oct 30 17:15:31.337657 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.337326    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-workload-socket" pod "16eaa2d1-45c5-4b91-8638-e6a5c492de5c" (UID: "16eaa2d1-45c5-4b91-8638-e6a5c492de5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:15:31.337751 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.337455    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-istio-envoy\") pod \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\" (UID: \"16eaa2d1-45c5-4b91-8638-e6a5c492de5c\") "
Oct 30 17:15:31.337751 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.337374    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-workload-certs" pod "16eaa2d1-45c5-4b91-8638-e6a5c492de5c" (UID: "16eaa2d1-45c5-4b91-8638-e6a5c492de5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:15:31.337751 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.337461    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-istiod-ca-cert" pod "16eaa2d1-45c5-4b91-8638-e6a5c492de5c" (UID: "16eaa2d1-45c5-4b91-8638-e6a5c492de5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:15:31.337751 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.337492    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-istio-data" pod "16eaa2d1-45c5-4b91-8638-e6a5c492de5c" (UID: "16eaa2d1-45c5-4b91-8638-e6a5c492de5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:15:31.338244 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.338224    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:15:31.338244 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.338244    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:15:31.338375 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.338254    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:15:31.338375 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.338263    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:15:31.338375 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.338271    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:15:31.339148 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.339128    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-kube-api-access-2gfkg" pod "16eaa2d1-45c5-4b91-8638-e6a5c492de5c" (UID: "16eaa2d1-45c5-4b91-8638-e6a5c492de5c"). InnerVolumeSpecName "kube-api-access-2gfkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:15:31.339637 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.339623    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-istio-podinfo" pod "16eaa2d1-45c5-4b91-8638-e6a5c492de5c" (UID: "16eaa2d1-45c5-4b91-8638-e6a5c492de5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:15:31.339803 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.339776    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-istio-envoy" pod "16eaa2d1-45c5-4b91-8638-e6a5c492de5c" (UID: "16eaa2d1-45c5-4b91-8638-e6a5c492de5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:15:31.339803 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.339783    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-istio-token" pod "16eaa2d1-45c5-4b91-8638-e6a5c492de5c" (UID: "16eaa2d1-45c5-4b91-8638-e6a5c492de5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:15:31.433555 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.433524    4433 generic.go:356] "Generic (PLEG): container finished" podID="16eaa2d1-45c5-4b91-8638-e6a5c492de5c" containerID="68d3078d10323e47f2fa32f2fbf0cfea638f5553493f1fa8d6de7dbc1b0adb0c" exitCode=0
Oct 30 17:15:31.433920 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.433575    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"
Oct 30 17:15:31.433920 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.433603    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s" event={"ID":"16eaa2d1-45c5-4b91-8638-e6a5c492de5c","Type":"ContainerDied","Data":"68d3078d10323e47f2fa32f2fbf0cfea638f5553493f1fa8d6de7dbc1b0adb0c"}
Oct 30 17:15:31.433920 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.433662    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s" event={"ID":"16eaa2d1-45c5-4b91-8638-e6a5c492de5c","Type":"ContainerDied","Data":"582572f6b8fd9a984ead60e19faf4da3193d98ce0f9e676f55861056c49801cb"}
Oct 30 17:15:31.433920 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.433693    4433 scope.go:122] "RemoveContainer" containerID="68d3078d10323e47f2fa32f2fbf0cfea638f5553493f1fa8d6de7dbc1b0adb0c"
Oct 30 17:15:31.439975 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.439944    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:15:31.439975 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.439976    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:15:31.440143 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.440035    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:15:31.440143 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.440050    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gfkg\" (UniqueName: \"kubernetes.io/projected/16eaa2d1-45c5-4b91-8638-e6a5c492de5c-kube-api-access-2gfkg\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:15:31.452537 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.446233    4433 scope.go:122] "RemoveContainer" containerID="68d3078d10323e47f2fa32f2fbf0cfea638f5553493f1fa8d6de7dbc1b0adb0c"
Oct 30 17:15:31.452537 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:15:31.446674    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68d3078d10323e47f2fa32f2fbf0cfea638f5553493f1fa8d6de7dbc1b0adb0c\": container with ID starting with 68d3078d10323e47f2fa32f2fbf0cfea638f5553493f1fa8d6de7dbc1b0adb0c not found: ID does not exist" containerID="68d3078d10323e47f2fa32f2fbf0cfea638f5553493f1fa8d6de7dbc1b0adb0c"
Oct 30 17:15:31.452537 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.446712    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68d3078d10323e47f2fa32f2fbf0cfea638f5553493f1fa8d6de7dbc1b0adb0c"} err="failed to get container status \"68d3078d10323e47f2fa32f2fbf0cfea638f5553493f1fa8d6de7dbc1b0adb0c\": rpc error: code = NotFound desc = could not find container \"68d3078d10323e47f2fa32f2fbf0cfea638f5553493f1fa8d6de7dbc1b0adb0c\": container with ID starting with 68d3078d10323e47f2fa32f2fbf0cfea638f5553493f1fa8d6de7dbc1b0adb0c not found: ID does not exist"
Oct 30 17:15:31.485027 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.484993    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"]
Oct 30 17:15:31.492197 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.492165    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-lmq9s"]
Oct 30 17:15:31.696681 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:31.696651    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16eaa2d1-45c5-4b91-8638-e6a5c492de5c" path="/var/lib/kubelet/pods/16eaa2d1-45c5-4b91-8638-e6a5c492de5c/volumes"
Oct 30 17:15:34.250760 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:34.250720    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk"]
Oct 30 17:15:34.251240 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:34.251043    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk" podUID="ae73bf58-d907-419a-b0c5-6d7b1b43e3f0" containerName="maas-api" containerID="cri-o://3e24937ccc5360db4e1c473abb90b1dbd1a221746a6a3ed8345b1d2f0f93353f" gracePeriod=30
Oct 30 17:15:34.920557 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:34.920521    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk"
Oct 30 17:15:34.999377 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:34.999336    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae73bf58-d907-419a-b0c5-6d7b1b43e3f0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae73bf58-d907-419a-b0c5-6d7b1b43e3f0-maas-api-tls\") pod \"ae73bf58-d907-419a-b0c5-6d7b1b43e3f0\" (UID: \"ae73bf58-d907-419a-b0c5-6d7b1b43e3f0\") "
Oct 30 17:15:34.999601 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:34.999583    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae73bf58-d907-419a-b0c5-6d7b1b43e3f0-kube-api-access-28kcj\" (UniqueName: \"kubernetes.io/projected/ae73bf58-d907-419a-b0c5-6d7b1b43e3f0-kube-api-access-28kcj\") pod \"ae73bf58-d907-419a-b0c5-6d7b1b43e3f0\" (UID: \"ae73bf58-d907-419a-b0c5-6d7b1b43e3f0\") "
Oct 30 17:15:34.999662 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:34.999630    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae73bf58-d907-419a-b0c5-6d7b1b43e3f0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae73bf58-d907-419a-b0c5-6d7b1b43e3f0-openshift-service-ca\") pod \"ae73bf58-d907-419a-b0c5-6d7b1b43e3f0\" (UID: \"ae73bf58-d907-419a-b0c5-6d7b1b43e3f0\") "
Oct 30 17:15:34.999915 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:34.999889    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae73bf58-d907-419a-b0c5-6d7b1b43e3f0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae73bf58-d907-419a-b0c5-6d7b1b43e3f0-metrics-certs\") pod \"ae73bf58-d907-419a-b0c5-6d7b1b43e3f0\" (UID: \"ae73bf58-d907-419a-b0c5-6d7b1b43e3f0\") "
Oct 30 17:15:35.000064 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:35.000033    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae73bf58-d907-419a-b0c5-6d7b1b43e3f0-openshift-service-ca" pod "ae73bf58-d907-419a-b0c5-6d7b1b43e3f0" (UID: "ae73bf58-d907-419a-b0c5-6d7b1b43e3f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:15:35.000502 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:35.000480    4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae73bf58-d907-419a-b0c5-6d7b1b43e3f0-openshift-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:15:35.001849 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:35.001821    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae73bf58-d907-419a-b0c5-6d7b1b43e3f0-maas-api-tls" pod "ae73bf58-d907-419a-b0c5-6d7b1b43e3f0" (UID: "ae73bf58-d907-419a-b0c5-6d7b1b43e3f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:15:35.002278 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:35.002256    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae73bf58-d907-419a-b0c5-6d7b1b43e3f0-metrics-certs" pod "ae73bf58-d907-419a-b0c5-6d7b1b43e3f0" (UID: "ae73bf58-d907-419a-b0c5-6d7b1b43e3f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:15:35.002479 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:35.002463    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae73bf58-d907-419a-b0c5-6d7b1b43e3f0-kube-api-access-28kcj" pod "ae73bf58-d907-419a-b0c5-6d7b1b43e3f0" (UID: "ae73bf58-d907-419a-b0c5-6d7b1b43e3f0"). InnerVolumeSpecName "kube-api-access-28kcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:15:35.105428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:35.105085    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae73bf58-d907-419a-b0c5-6d7b1b43e3f0-maas-api-tls\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:15:35.105428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:35.105127    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28kcj\" (UniqueName: \"kubernetes.io/projected/ae73bf58-d907-419a-b0c5-6d7b1b43e3f0-kube-api-access-28kcj\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:15:35.105428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:35.105141    4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae73bf58-d907-419a-b0c5-6d7b1b43e3f0-metrics-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:15:35.467249 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:35.467217    4433 generic.go:356] "Generic (PLEG): container finished" podID="ae73bf58-d907-419a-b0c5-6d7b1b43e3f0" containerID="3e24937ccc5360db4e1c473abb90b1dbd1a221746a6a3ed8345b1d2f0f93353f" exitCode=0
Oct 30 17:15:35.467907 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:35.467282    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk"
Oct 30 17:15:35.467907 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:35.467299    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk" event={"ID":"ae73bf58-d907-419a-b0c5-6d7b1b43e3f0","Type":"ContainerDied","Data":"3e24937ccc5360db4e1c473abb90b1dbd1a221746a6a3ed8345b1d2f0f93353f"}
Oct 30 17:15:35.467907 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:35.467339    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk" event={"ID":"ae73bf58-d907-419a-b0c5-6d7b1b43e3f0","Type":"ContainerDied","Data":"7f69f49a4ba87314bf66d1104b7c21cd8079d813252abb749ddf18015b2dba61"}
Oct 30 17:15:35.467907 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:35.467360    4433 scope.go:122] "RemoveContainer" containerID="3e24937ccc5360db4e1c473abb90b1dbd1a221746a6a3ed8345b1d2f0f93353f"
Oct 30 17:15:35.478662 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:35.478634    4433 scope.go:122] "RemoveContainer" containerID="3e24937ccc5360db4e1c473abb90b1dbd1a221746a6a3ed8345b1d2f0f93353f"
Oct 30 17:15:35.478947 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:15:35.478929    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e24937ccc5360db4e1c473abb90b1dbd1a221746a6a3ed8345b1d2f0f93353f\": container with ID starting with 3e24937ccc5360db4e1c473abb90b1dbd1a221746a6a3ed8345b1d2f0f93353f not found: ID does not exist" containerID="3e24937ccc5360db4e1c473abb90b1dbd1a221746a6a3ed8345b1d2f0f93353f"
Oct 30 17:15:35.479013 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:35.478956    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e24937ccc5360db4e1c473abb90b1dbd1a221746a6a3ed8345b1d2f0f93353f"} err="failed to get container status \"3e24937ccc5360db4e1c473abb90b1dbd1a221746a6a3ed8345b1d2f0f93353f\": rpc error: code = NotFound desc = could not find container \"3e24937ccc5360db4e1c473abb90b1dbd1a221746a6a3ed8345b1d2f0f93353f\": container with ID starting with 3e24937ccc5360db4e1c473abb90b1dbd1a221746a6a3ed8345b1d2f0f93353f not found: ID does not exist"
Oct 30 17:15:35.546486 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:35.546447    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk"]
Oct 30 17:15:35.557678 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:35.557384    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-728ad7-898484f7d-9kpsk"]
Oct 30 17:15:35.696267 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:35.696232    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae73bf58-d907-419a-b0c5-6d7b1b43e3f0" path="/var/lib/kubelet/pods/ae73bf58-d907-419a-b0c5-6d7b1b43e3f0/volumes"
Oct 30 17:15:41.803051 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:41.803015    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:15:43.974079 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:43.974039    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2"]
Oct 30 17:15:43.974770 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:43.974750    4433 cpu_manager.go:414] "containerMap: removing container" podUID="de736fdc-db4a-4b35-b623-5a037e0579d6" containerName="cleanup"
Oct 30 17:15:43.974838 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:43.974771    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de736fdc-db4a-4b35-b623-5a037e0579d6" containerName="cleanup"
Oct 30 17:15:43.974838 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:43.974798    4433 cpu_manager.go:414] "containerMap: removing container" podUID="4e355962-aa3b-4c35-8f1e-b98f41716880" containerName="cleanup"
Oct 30 17:15:43.974838 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:43.974803    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e355962-aa3b-4c35-8f1e-b98f41716880" containerName="cleanup"
Oct 30 17:15:43.974838 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:43.974822    4433 cpu_manager.go:414] "containerMap: removing container" podUID="16eaa2d1-45c5-4b91-8638-e6a5c492de5c" containerName="istio-proxy"
Oct 30 17:15:43.974838 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:43.974828    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16eaa2d1-45c5-4b91-8638-e6a5c492de5c" containerName="istio-proxy"
Oct 30 17:15:43.975010 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:43.974840    4433 cpu_manager.go:414] "containerMap: removing container" podUID="ae73bf58-d907-419a-b0c5-6d7b1b43e3f0" containerName="maas-api"
Oct 30 17:15:43.975010 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:43.974845    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae73bf58-d907-419a-b0c5-6d7b1b43e3f0" containerName="maas-api"
Oct 30 17:15:43.975010 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:43.974861    4433 cpu_manager.go:414] "containerMap: removing container" podUID="b0ab47a4-a873-43e7-b528-bc733d58e7a7" containerName="cleanup"
Oct 30 17:15:43.975010 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:43.974865    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ab47a4-a873-43e7-b528-bc733d58e7a7" containerName="cleanup"
Oct 30 17:15:43.975010 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:43.974875    4433 cpu_manager.go:414] "containerMap: removing container" podUID="938091c7-0d73-4944-a33b-f123c256a2a4" containerName="cleanup"
Oct 30 17:15:43.975010 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:43.974879    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="938091c7-0d73-4944-a33b-f123c256a2a4" containerName="cleanup"
Oct 30 17:15:43.975010 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:43.974961    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e355962-aa3b-4c35-8f1e-b98f41716880" containerName="cleanup"
Oct 30 17:15:43.975010 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:43.974971    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="16eaa2d1-45c5-4b91-8638-e6a5c492de5c" containerName="istio-proxy"
Oct 30 17:15:43.975010 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:43.974978    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aa1ca72-36df-4d56-b3f2-e7b33f4e1379" containerName="cleanup"
Oct 30 17:15:43.975010 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:43.974984    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="c34089da-9f1d-461a-a3ff-65b5d894eef4" containerName="cleanup"
Oct 30 17:15:43.975010 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:43.974991    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae73bf58-d907-419a-b0c5-6d7b1b43e3f0" containerName="maas-api"
Oct 30 17:15:43.975010 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:43.974999    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="de736fdc-db4a-4b35-b623-5a037e0579d6" containerName="cleanup"
Oct 30 17:15:43.975010 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:43.975006    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff2fdc73-1d9a-4bdf-9ee6-63d37e255d9b" containerName="cleanup"
Oct 30 17:15:43.975010 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:43.975013    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="b78c3100-b1b6-4204-af12-54870e4b8dbb" containerName="cleanup"
Oct 30 17:15:43.975502 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:43.975031    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ab47a4-a873-43e7-b528-bc733d58e7a7" containerName="cleanup"
Oct 30 17:15:43.980627 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:43.980605    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2"
Oct 30 17:15:44.005281 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:44.004698    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2"]
Oct 30 17:15:44.010861 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:44.010382    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-5d67ca-serving-cert\""
Oct 30 17:15:44.110589 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:44.110517    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc12f255-74a5-4c88-b67b-a0c377797e84-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2\" (UID: \"fc12f255-74a5-4c88-b67b-a0c377797e84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2"
Oct 30 17:15:44.110589 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:44.110576    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc12f255-74a5-4c88-b67b-a0c377797e84-metrics-certs\") pod \"maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2\" (UID: \"fc12f255-74a5-4c88-b67b-a0c377797e84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2"
Oct 30 17:15:44.110852 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:44.110710    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfpq2\" (UniqueName: \"kubernetes.io/projected/fc12f255-74a5-4c88-b67b-a0c377797e84-kube-api-access-xfpq2\") pod \"maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2\" (UID: \"fc12f255-74a5-4c88-b67b-a0c377797e84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2"
Oct 30 17:15:44.110921 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:44.110873    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc12f255-74a5-4c88-b67b-a0c377797e84-maas-api-tls\") pod \"maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2\" (UID: \"fc12f255-74a5-4c88-b67b-a0c377797e84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2"
Oct 30 17:15:44.227192 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:44.227096    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc12f255-74a5-4c88-b67b-a0c377797e84-maas-api-tls\") pod \"maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2\" (UID: \"fc12f255-74a5-4c88-b67b-a0c377797e84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2"
Oct 30 17:15:44.240493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:44.230678    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc12f255-74a5-4c88-b67b-a0c377797e84-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2\" (UID: \"fc12f255-74a5-4c88-b67b-a0c377797e84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2"
Oct 30 17:15:44.240493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:44.230738    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc12f255-74a5-4c88-b67b-a0c377797e84-metrics-certs\") pod \"maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2\" (UID: \"fc12f255-74a5-4c88-b67b-a0c377797e84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2"
Oct 30 17:15:44.240493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:44.230910    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfpq2\" (UniqueName: \"kubernetes.io/projected/fc12f255-74a5-4c88-b67b-a0c377797e84-kube-api-access-xfpq2\") pod \"maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2\" (UID: \"fc12f255-74a5-4c88-b67b-a0c377797e84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2"
Oct 30 17:15:44.240493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:44.232358    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc12f255-74a5-4c88-b67b-a0c377797e84-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2\" (UID: \"fc12f255-74a5-4c88-b67b-a0c377797e84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2"
Oct 30 17:15:44.248080 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:44.248047    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc12f255-74a5-4c88-b67b-a0c377797e84-metrics-certs\") pod \"maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2\" (UID: \"fc12f255-74a5-4c88-b67b-a0c377797e84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2"
Oct 30 17:15:44.255977 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:44.255908    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfpq2\" (UniqueName: \"kubernetes.io/projected/fc12f255-74a5-4c88-b67b-a0c377797e84-kube-api-access-xfpq2\") pod \"maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2\" (UID: \"fc12f255-74a5-4c88-b67b-a0c377797e84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2"
Oct 30 17:15:44.294830 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:44.294788    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc12f255-74a5-4c88-b67b-a0c377797e84-maas-api-tls\") pod \"maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2\" (UID: \"fc12f255-74a5-4c88-b67b-a0c377797e84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2"
Oct 30 17:15:44.296988 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:44.296764    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2"
Oct 30 17:15:44.699795 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:44.684214    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2"]
Oct 30 17:15:44.702887 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:15:44.702845    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc12f255_74a5_4c88_b67b_a0c377797e84.slice/crio-6835d785494154bae86a6d5dac4714c4f139266e037af579dcc336a14b54da67 WatchSource:0}: Error finding container 6835d785494154bae86a6d5dac4714c4f139266e037af579dcc336a14b54da67: Status 404 returned error can't find the container with id 6835d785494154bae86a6d5dac4714c4f139266e037af579dcc336a14b54da67
Oct 30 17:15:45.622533 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:45.622485    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2" event={"ID":"fc12f255-74a5-4c88-b67b-a0c377797e84","Type":"ContainerStarted","Data":"f14b17f8f6814cc9e0771490a8163284320bf592c1137a85c50f5c565e6cf340"}
Oct 30 17:15:45.622963 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:45.622559    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2" event={"ID":"fc12f255-74a5-4c88-b67b-a0c377797e84","Type":"ContainerStarted","Data":"6835d785494154bae86a6d5dac4714c4f139266e037af579dcc336a14b54da67"}
Oct 30 17:15:45.622963 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:45.622764    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2"
Oct 30 17:15:45.668357 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:45.667008    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2" podStartSLOduration=2.350483622 podStartE2EDuration="2.66698811s" totalImagesPullingTime="316.504488ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:45.663926027 +0000 UTC m=+1722.098828234" watchObservedRunningTime="2026-10-30 17:15:45.66698811 +0000 UTC m=+1722.101890322"
Oct 30 17:15:51.639817 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:51.639783    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2"
Oct 30 17:15:57.093635 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:15:57.093595    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:15:59.170868 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.170830    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"]
Oct 30 17:15:59.175224 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.175198    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"
Oct 30 17:15:59.228626 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.228587    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"]
Oct 30 17:15:59.246927 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.246893    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7c52f8-openshift-default-dockercfg-lmxqw\""
Oct 30 17:15:59.277105 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.277064    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q57kp\" (UniqueName: \"kubernetes.io/projected/f265af5b-6027-4101-99db-328a61fd85da-kube-api-access-q57kp\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"
Oct 30 17:15:59.277312 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.277165    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f265af5b-6027-4101-99db-328a61fd85da-istio-podinfo\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"
Oct 30 17:15:59.277312 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.277208    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f265af5b-6027-4101-99db-328a61fd85da-istiod-ca-cert\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"
Oct 30 17:15:59.277312 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.277297    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f265af5b-6027-4101-99db-328a61fd85da-istio-data\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"
Oct 30 17:15:59.277527 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.277379    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f265af5b-6027-4101-99db-328a61fd85da-workload-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"
Oct 30 17:15:59.277527 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.277394    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f265af5b-6027-4101-99db-328a61fd85da-istio-envoy\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"
Oct 30 17:15:59.277662 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.277588    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f265af5b-6027-4101-99db-328a61fd85da-credential-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"
Oct 30 17:15:59.277662 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.277643    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f265af5b-6027-4101-99db-328a61fd85da-istio-token\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"
Oct 30 17:15:59.277765 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.277677    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f265af5b-6027-4101-99db-328a61fd85da-workload-certs\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"
Oct 30 17:15:59.378554 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.378520    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f265af5b-6027-4101-99db-328a61fd85da-istio-data\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"
Oct 30 17:15:59.378769 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.378616    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f265af5b-6027-4101-99db-328a61fd85da-workload-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"
Oct 30 17:15:59.378769 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.378643    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f265af5b-6027-4101-99db-328a61fd85da-istio-envoy\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"
Oct 30 17:15:59.378893 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.378767    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f265af5b-6027-4101-99db-328a61fd85da-credential-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"
Oct 30 17:15:59.378893 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.378822    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f265af5b-6027-4101-99db-328a61fd85da-istio-token\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"
Oct 30 17:15:59.378893 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.378869    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f265af5b-6027-4101-99db-328a61fd85da-workload-certs\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"
Oct 30 17:15:59.379066 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.378948    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q57kp\" (UniqueName: \"kubernetes.io/projected/f265af5b-6027-4101-99db-328a61fd85da-kube-api-access-q57kp\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"
Oct 30 17:15:59.379066 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.378998    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f265af5b-6027-4101-99db-328a61fd85da-istio-data\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"
Oct 30 17:15:59.379066 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.379005    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f265af5b-6027-4101-99db-328a61fd85da-istio-podinfo\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"
Oct 30 17:15:59.379066 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.379054    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f265af5b-6027-4101-99db-328a61fd85da-workload-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"
Oct 30 17:15:59.379288 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.379064    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f265af5b-6027-4101-99db-328a61fd85da-istiod-ca-cert\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"
Oct 30 17:15:59.379374 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.379349    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f265af5b-6027-4101-99db-328a61fd85da-credential-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"
Oct 30 17:15:59.379480 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.379460    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f265af5b-6027-4101-99db-328a61fd85da-workload-certs\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"
Oct 30 17:15:59.379750 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.379728    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f265af5b-6027-4101-99db-328a61fd85da-istiod-ca-cert\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"
Oct 30 17:15:59.381450 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.381430    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f265af5b-6027-4101-99db-328a61fd85da-istio-envoy\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"
Oct 30 17:15:59.381625 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.381606    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f265af5b-6027-4101-99db-328a61fd85da-istio-podinfo\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"
Oct 30 17:15:59.422020 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.421941    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f265af5b-6027-4101-99db-328a61fd85da-istio-token\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"
Oct 30 17:15:59.422320 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.422301    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q57kp\" (UniqueName: \"kubernetes.io/projected/f265af5b-6027-4101-99db-328a61fd85da-kube-api-access-q57kp\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"
Oct 30 17:15:59.493525 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.493487    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"
Oct 30 17:15:59.819171 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.819143    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"]
Oct 30 17:15:59.821366 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:15:59.821207    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf265af5b_6027_4101_99db_328a61fd85da.slice/crio-1217aad569e894409c0885012fce40ea9894e7e8efba1ad96fdea9b68585c700 WatchSource:0}: Error finding container 1217aad569e894409c0885012fce40ea9894e7e8efba1ad96fdea9b68585c700: Status 404 returned error can't find the container with id 1217aad569e894409c0885012fce40ea9894e7e8efba1ad96fdea9b68585c700
Oct 30 17:15:59.824363 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.824330    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:15:59.824487 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.824397    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:15:59.824487 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.824444    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:15:59.835956 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:15:59.835921    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm" event={"ID":"f265af5b-6027-4101-99db-328a61fd85da","Type":"ContainerStarted","Data":"1217aad569e894409c0885012fce40ea9894e7e8efba1ad96fdea9b68585c700"}
Oct 30 17:16:00.843532 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:00.843495    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm" event={"ID":"f265af5b-6027-4101-99db-328a61fd85da","Type":"ContainerStarted","Data":"524e209f4c21b9a4f87668ad8294bb6500f9b55a7fcc506c739bbada9d73c4fd"}
Oct 30 17:16:00.909072 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:00.909016    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm" podStartSLOduration=1.908997555 podStartE2EDuration="1.908997555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:00.905712792 +0000 UTC m=+1737.340615012" watchObservedRunningTime="2026-10-30 17:16:00.908997555 +0000 UTC m=+1737.343899762"
Oct 30 17:16:01.494001 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:01.493956    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"
Oct 30 17:16:01.495890 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:01.495844    4433 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 17:16:01.496065 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:01.495928    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm" podUID="f265af5b-6027-4101-99db-328a61fd85da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 17:16:02.494989 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:02.494030    4433 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 17:16:02.494989 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:02.494151    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm" podUID="f265af5b-6027-4101-99db-328a61fd85da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 17:16:03.494017 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:03.493976    4433 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 17:16:03.494191 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:03.494037    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm" podUID="f265af5b-6027-4101-99db-328a61fd85da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 17:16:04.493968 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:04.493920    4433 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 17:16:04.494451 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:04.493979    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm" podUID="f265af5b-6027-4101-99db-328a61fd85da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 17:16:05.493887 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:05.493851    4433 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 17:16:05.494074 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:05.493905    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm" podUID="f265af5b-6027-4101-99db-328a61fd85da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 17:16:06.606552 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:06.606517    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"
Oct 30 17:16:06.608073 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:06.608048    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"
Oct 30 17:16:06.609226 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:06.609196    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"
Oct 30 17:16:24.875900 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:24.875859    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"]
Oct 30 17:16:24.876618 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:24.876135    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr" podUID="1ee372e3-661e-49b0-ae79-6c7ae7736955" containerName="istio-proxy" containerID="cri-o://467873d1735e34e1aafdc99c7f54ef4e8fe7e4982229d10050932dd70f3b808f" gracePeriod=30
Oct 30 17:16:26.979273 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:16:26.979240    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:16:30.161958 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:30.161860    4433 generic.go:356] "Generic (PLEG): container finished" podID="1ee372e3-661e-49b0-ae79-6c7ae7736955" containerID="467873d1735e34e1aafdc99c7f54ef4e8fe7e4982229d10050932dd70f3b808f" exitCode=0
Oct 30 17:16:30.162337 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:30.161950    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr" event={"ID":"1ee372e3-661e-49b0-ae79-6c7ae7736955","Type":"ContainerDied","Data":"467873d1735e34e1aafdc99c7f54ef4e8fe7e4982229d10050932dd70f3b808f"}
Oct 30 17:16:30.209084 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:30.209054    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"
Oct 30 17:16:30.315605 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:30.315563    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ee372e3-661e-49b0-ae79-6c7ae7736955-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee372e3-661e-49b0-ae79-6c7ae7736955-workload-socket\") pod \"1ee372e3-661e-49b0-ae79-6c7ae7736955\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") "
Oct 30 17:16:30.315814 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:30.315631    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ee372e3-661e-49b0-ae79-6c7ae7736955-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ee372e3-661e-49b0-ae79-6c7ae7736955-istio-podinfo\") pod \"1ee372e3-661e-49b0-ae79-6c7ae7736955\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") "
Oct 30 17:16:30.315814 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:30.315739    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ee372e3-661e-49b0-ae79-6c7ae7736955-kube-api-access-bgxzv\" (UniqueName: \"kubernetes.io/projected/1ee372e3-661e-49b0-ae79-6c7ae7736955-kube-api-access-bgxzv\") pod \"1ee372e3-661e-49b0-ae79-6c7ae7736955\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") "
Oct 30 17:16:30.316175 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:30.315771    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ee372e3-661e-49b0-ae79-6c7ae7736955-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ee372e3-661e-49b0-ae79-6c7ae7736955-istio-envoy\") pod \"1ee372e3-661e-49b0-ae79-6c7ae7736955\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") "
Oct 30 17:16:30.316461 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:30.316383    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee372e3-661e-49b0-ae79-6c7ae7736955-workload-socket" pod "1ee372e3-661e-49b0-ae79-6c7ae7736955" (UID: "1ee372e3-661e-49b0-ae79-6c7ae7736955"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:16:30.316653 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:30.316513    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ee372e3-661e-49b0-ae79-6c7ae7736955-istio-token\" (UniqueName: \"kubernetes.io/projected/1ee372e3-661e-49b0-ae79-6c7ae7736955-istio-token\") pod \"1ee372e3-661e-49b0-ae79-6c7ae7736955\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") "
Oct 30 17:16:30.317168 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:30.317130    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee372e3-661e-49b0-ae79-6c7ae7736955-workload-certs" pod "1ee372e3-661e-49b0-ae79-6c7ae7736955" (UID: "1ee372e3-661e-49b0-ae79-6c7ae7736955"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:16:30.317318 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:30.317299    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ee372e3-661e-49b0-ae79-6c7ae7736955-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ee372e3-661e-49b0-ae79-6c7ae7736955-workload-certs\") pod \"1ee372e3-661e-49b0-ae79-6c7ae7736955\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") "
Oct 30 17:16:30.317530 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:30.317511    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ee372e3-661e-49b0-ae79-6c7ae7736955-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee372e3-661e-49b0-ae79-6c7ae7736955-credential-socket\") pod \"1ee372e3-661e-49b0-ae79-6c7ae7736955\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") "
Oct 30 17:16:30.318011 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:30.317841    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee372e3-661e-49b0-ae79-6c7ae7736955-credential-socket" pod "1ee372e3-661e-49b0-ae79-6c7ae7736955" (UID: "1ee372e3-661e-49b0-ae79-6c7ae7736955"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:16:30.318251 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:30.318233    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ee372e3-661e-49b0-ae79-6c7ae7736955-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ee372e3-661e-49b0-ae79-6c7ae7736955-istio-data\") pod \"1ee372e3-661e-49b0-ae79-6c7ae7736955\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") "
Oct 30 17:16:30.318704 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:30.318577    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee372e3-661e-49b0-ae79-6c7ae7736955-istio-data" pod "1ee372e3-661e-49b0-ae79-6c7ae7736955" (UID: "1ee372e3-661e-49b0-ae79-6c7ae7736955"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:16:30.318917 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:30.318897    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ee372e3-661e-49b0-ae79-6c7ae7736955-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ee372e3-661e-49b0-ae79-6c7ae7736955-istiod-ca-cert\") pod \"1ee372e3-661e-49b0-ae79-6c7ae7736955\" (UID: \"1ee372e3-661e-49b0-ae79-6c7ae7736955\") "
Oct 30 17:16:30.319264 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:30.319239    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ee372e3-661e-49b0-ae79-6c7ae7736955-istiod-ca-cert" pod "1ee372e3-661e-49b0-ae79-6c7ae7736955" (UID: "1ee372e3-661e-49b0-ae79-6c7ae7736955"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:16:30.319414 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:30.319381    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ee372e3-661e-49b0-ae79-6c7ae7736955-istio-token" pod "1ee372e3-661e-49b0-ae79-6c7ae7736955" (UID: "1ee372e3-661e-49b0-ae79-6c7ae7736955"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:16:30.320061 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:30.320032    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee372e3-661e-49b0-ae79-6c7ae7736955-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:16:30.320160 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:30.320065    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ee372e3-661e-49b0-ae79-6c7ae7736955-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:16:30.320160 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:30.320080    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ee372e3-661e-49b0-ae79-6c7ae7736955-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:16:30.320160 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:30.320095    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee372e3-661e-49b0-ae79-6c7ae7736955-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:16:30.320160 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:30.320108    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ee372e3-661e-49b0-ae79-6c7ae7736955-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:16:30.320160 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:30.320121    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ee372e3-661e-49b0-ae79-6c7ae7736955-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:16:30.320575 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:30.320543    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ee372e3-661e-49b0-ae79-6c7ae7736955-istio-podinfo" pod "1ee372e3-661e-49b0-ae79-6c7ae7736955" (UID: "1ee372e3-661e-49b0-ae79-6c7ae7736955"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:16:30.320692 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:30.320575    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ee372e3-661e-49b0-ae79-6c7ae7736955-kube-api-access-bgxzv" pod "1ee372e3-661e-49b0-ae79-6c7ae7736955" (UID: "1ee372e3-661e-49b0-ae79-6c7ae7736955"). InnerVolumeSpecName "kube-api-access-bgxzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:16:30.322333 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:30.322310    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee372e3-661e-49b0-ae79-6c7ae7736955-istio-envoy" pod "1ee372e3-661e-49b0-ae79-6c7ae7736955" (UID: "1ee372e3-661e-49b0-ae79-6c7ae7736955"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:16:30.422086 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:30.421999    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ee372e3-661e-49b0-ae79-6c7ae7736955-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:16:30.422086 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:30.422034    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgxzv\" (UniqueName: \"kubernetes.io/projected/1ee372e3-661e-49b0-ae79-6c7ae7736955-kube-api-access-bgxzv\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:16:30.422086 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:30.422045    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ee372e3-661e-49b0-ae79-6c7ae7736955-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:16:31.170734 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:31.170706    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"
Oct 30 17:16:31.171188 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:31.170705    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr" event={"ID":"1ee372e3-661e-49b0-ae79-6c7ae7736955","Type":"ContainerDied","Data":"90bc589bcb024c72592857bf72487ac00008930c939447e4e8b33290a03a976f"}
Oct 30 17:16:31.171188 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:31.170838    4433 scope.go:122] "RemoveContainer" containerID="467873d1735e34e1aafdc99c7f54ef4e8fe7e4982229d10050932dd70f3b808f"
Oct 30 17:16:31.238339 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:31.238304    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"]
Oct 30 17:16:31.243462 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:31.243426    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-5zfsr"]
Oct 30 17:16:31.615152 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:31.615105    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:16:31.703033 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:31.702997    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ee372e3-661e-49b0-ae79-6c7ae7736955" path="/var/lib/kubelet/pods/1ee372e3-661e-49b0-ae79-6c7ae7736955/volumes"
Oct 30 17:16:36.699976 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:36.698213    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7"]
Oct 30 17:16:36.699976 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:36.699240    4433 cpu_manager.go:414] "containerMap: removing container" podUID="1ee372e3-661e-49b0-ae79-6c7ae7736955" containerName="istio-proxy"
Oct 30 17:16:36.699976 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:36.699264    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ee372e3-661e-49b0-ae79-6c7ae7736955" containerName="istio-proxy"
Oct 30 17:16:36.699976 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:36.699442    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ee372e3-661e-49b0-ae79-6c7ae7736955" containerName="istio-proxy"
Oct 30 17:16:36.706040 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:36.706011    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7"
Oct 30 17:16:36.722974 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:36.722943    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7"]
Oct 30 17:16:36.729987 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:36.729959    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d003a5\"/\"openshift-service-ca.crt\""
Oct 30 17:16:36.730205 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:36.729998    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d003a5\"/\"kube-root-ca.crt\""
Oct 30 17:16:36.733285 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:36.733260    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d003a5\"/\"default-dockercfg-c5vpk\""
Oct 30 17:16:36.733399 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:36.733299    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d003a5\"/\"test-model-d003a5-kserve-self-signed-certs\""
Oct 30 17:16:36.856789 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:36.856745    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-tmp-dir\") pod \"test-model-d003a5-kserve-5dddbf7989-pd4b7\" (UID: \"4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe\") " pod="ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7"
Oct 30 17:16:36.857001 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:36.856829    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77nl4\" (UniqueName: \"kubernetes.io/projected/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-kube-api-access-77nl4\") pod \"test-model-d003a5-kserve-5dddbf7989-pd4b7\" (UID: \"4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe\") " pod="ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7"
Oct 30 17:16:36.857001 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:36.856958    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-tls-certs\") pod \"test-model-d003a5-kserve-5dddbf7989-pd4b7\" (UID: \"4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe\") " pod="ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7"
Oct 30 17:16:36.857130 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:36.857087    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-model-cache\") pod \"test-model-d003a5-kserve-5dddbf7989-pd4b7\" (UID: \"4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe\") " pod="ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7"
Oct 30 17:16:36.857312 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:36.857290    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-home\") pod \"test-model-d003a5-kserve-5dddbf7989-pd4b7\" (UID: \"4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe\") " pod="ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7"
Oct 30 17:16:36.857388 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:36.857346    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-dshm\") pod \"test-model-d003a5-kserve-5dddbf7989-pd4b7\" (UID: \"4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe\") " pod="ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7"
Oct 30 17:16:36.958955 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:36.958862    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-model-cache\") pod \"test-model-d003a5-kserve-5dddbf7989-pd4b7\" (UID: \"4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe\") " pod="ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7"
Oct 30 17:16:36.959159 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:36.958982    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-home\") pod \"test-model-d003a5-kserve-5dddbf7989-pd4b7\" (UID: \"4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe\") " pod="ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7"
Oct 30 17:16:36.959159 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:36.959012    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-dshm\") pod \"test-model-d003a5-kserve-5dddbf7989-pd4b7\" (UID: \"4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe\") " pod="ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7"
Oct 30 17:16:36.959159 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:36.959041    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-tmp-dir\") pod \"test-model-d003a5-kserve-5dddbf7989-pd4b7\" (UID: \"4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe\") " pod="ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7"
Oct 30 17:16:36.959159 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:36.959083    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77nl4\" (UniqueName: \"kubernetes.io/projected/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-kube-api-access-77nl4\") pod \"test-model-d003a5-kserve-5dddbf7989-pd4b7\" (UID: \"4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe\") " pod="ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7"
Oct 30 17:16:36.959394 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:36.959166    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-tls-certs\") pod \"test-model-d003a5-kserve-5dddbf7989-pd4b7\" (UID: \"4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe\") " pod="ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7"
Oct 30 17:16:36.959394 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:36.959345    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-model-cache\") pod \"test-model-d003a5-kserve-5dddbf7989-pd4b7\" (UID: \"4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe\") " pod="ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7"
Oct 30 17:16:36.959394 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:36.959351    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-home\") pod \"test-model-d003a5-kserve-5dddbf7989-pd4b7\" (UID: \"4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe\") " pod="ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7"
Oct 30 17:16:36.959654 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:36.959631    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-tmp-dir\") pod \"test-model-d003a5-kserve-5dddbf7989-pd4b7\" (UID: \"4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe\") " pod="ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7"
Oct 30 17:16:36.961651 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:36.961623    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-dshm\") pod \"test-model-d003a5-kserve-5dddbf7989-pd4b7\" (UID: \"4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe\") " pod="ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7"
Oct 30 17:16:36.961787 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:36.961692    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-tls-certs\") pod \"test-model-d003a5-kserve-5dddbf7989-pd4b7\" (UID: \"4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe\") " pod="ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7"
Oct 30 17:16:37.011781 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:37.011745    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77nl4\" (UniqueName: \"kubernetes.io/projected/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-kube-api-access-77nl4\") pod \"test-model-d003a5-kserve-5dddbf7989-pd4b7\" (UID: \"4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe\") " pod="ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7"
Oct 30 17:16:37.020435 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:37.020395    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7"
Oct 30 17:16:37.732579 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:37.732420    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7"]
Oct 30 17:16:38.257200 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:38.257158    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7" event={"ID":"4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe","Type":"ContainerStarted","Data":"0ceb38e3f33ceff1771d1c8e56d54334b13322f739818972d787e197c5d98c46"}
Oct 30 17:16:38.257467 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:38.257213    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7" event={"ID":"4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe","Type":"ContainerStarted","Data":"62b32f981aca20cc1d896a7245270c000b57820fdc5fc3f8da751a0b2fd5f50a"}
Oct 30 17:16:39.020610 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:39.020574    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7"
Oct 30 17:16:39.880850 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:39.880791    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7" podStartSLOduration=3.880775018 podStartE2EDuration="3.880775018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:38.310847593 +0000 UTC m=+1774.745749800" watchObservedRunningTime="2026-10-30 17:16:39.880775018 +0000 UTC m=+1776.315677224"
Oct 30 17:16:39.884283 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:39.884250    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"]
Oct 30 17:16:39.884525 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:39.884499    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp" podUID="47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7" containerName="istio-proxy" containerID="cri-o://06f4bf8086cc8a605ea07e6b6360b079494cd179289e0c849ea2eb0c0c62f5c7" gracePeriod=30
Oct 30 17:16:45.162840 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.162770    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"
Oct 30 17:16:45.239171 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.239140    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-credential-socket\") pod \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") "
Oct 30 17:16:45.239372 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.239179    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-workload-certs\") pod \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") "
Oct 30 17:16:45.239372 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.239201    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-workload-socket\") pod \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") "
Oct 30 17:16:45.239372 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.239233    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-istio-token\" (UniqueName: \"kubernetes.io/projected/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-istio-token\") pod \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") "
Oct 30 17:16:45.239603 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.239483    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-istiod-ca-cert\") pod \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") "
Oct 30 17:16:45.239603 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.239558    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-istio-envoy\") pod \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") "
Oct 30 17:16:45.239603 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.239549    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-credential-socket" pod "47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7" (UID: "47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:16:45.240079 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.239608    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-kube-api-access-d4h8h\" (UniqueName: \"kubernetes.io/projected/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-kube-api-access-d4h8h\") pod \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") "
Oct 30 17:16:45.240079 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.239711    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-istio-data\") pod \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") "
Oct 30 17:16:45.240079 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.239867    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-istio-podinfo\") pod \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\" (UID: \"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7\") "
Oct 30 17:16:45.240079 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.239900    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-istiod-ca-cert" pod "47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7" (UID: "47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:16:45.240865 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.240841    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:16:45.241018 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.240869    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:16:45.241018 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.240973    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-workload-socket" pod "47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7" (UID: "47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:16:45.241145 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.241013    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-workload-certs" pod "47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7" (UID: "47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:16:45.242895 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.241587    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-istio-data" pod "47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7" (UID: "47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:16:45.243355 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.243323    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-istio-token" pod "47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7" (UID: "47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:16:45.243630 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.243599    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-kube-api-access-d4h8h" pod "47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7" (UID: "47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7"). InnerVolumeSpecName "kube-api-access-d4h8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:16:45.246599 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.246562    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-istio-envoy" pod "47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7" (UID: "47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:16:45.249849 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.249813    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-istio-podinfo" pod "47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7" (UID: "47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:16:45.307148 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.307108    4433 generic.go:356] "Generic (PLEG): container finished" podID="47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7" containerID="06f4bf8086cc8a605ea07e6b6360b079494cd179289e0c849ea2eb0c0c62f5c7" exitCode=0
Oct 30 17:16:45.307548 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.307162    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"
Oct 30 17:16:45.307548 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.307194    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp" event={"ID":"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7","Type":"ContainerDied","Data":"06f4bf8086cc8a605ea07e6b6360b079494cd179289e0c849ea2eb0c0c62f5c7"}
Oct 30 17:16:45.307548 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.307262    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp" event={"ID":"47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7","Type":"ContainerDied","Data":"94884f34f91a81fb98447cc92ea53c0d572c2e0c99756156c261f9948b147f4f"}
Oct 30 17:16:45.307548 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.307293    4433 scope.go:122] "RemoveContainer" containerID="06f4bf8086cc8a605ea07e6b6360b079494cd179289e0c849ea2eb0c0c62f5c7"
Oct 30 17:16:45.318756 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.318724    4433 scope.go:122] "RemoveContainer" containerID="06f4bf8086cc8a605ea07e6b6360b079494cd179289e0c849ea2eb0c0c62f5c7"
Oct 30 17:16:45.319102 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:16:45.319079    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06f4bf8086cc8a605ea07e6b6360b079494cd179289e0c849ea2eb0c0c62f5c7\": container with ID starting with 06f4bf8086cc8a605ea07e6b6360b079494cd179289e0c849ea2eb0c0c62f5c7 not found: ID does not exist" containerID="06f4bf8086cc8a605ea07e6b6360b079494cd179289e0c849ea2eb0c0c62f5c7"
Oct 30 17:16:45.319200 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.319112    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06f4bf8086cc8a605ea07e6b6360b079494cd179289e0c849ea2eb0c0c62f5c7"} err="failed to get container status \"06f4bf8086cc8a605ea07e6b6360b079494cd179289e0c849ea2eb0c0c62f5c7\": rpc error: code = NotFound desc = could not find container \"06f4bf8086cc8a605ea07e6b6360b079494cd179289e0c849ea2eb0c0c62f5c7\": container with ID starting with 06f4bf8086cc8a605ea07e6b6360b079494cd179289e0c849ea2eb0c0c62f5c7 not found: ID does not exist"
Oct 30 17:16:45.342904 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.342871    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:16:45.343198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.343162    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:16:45.343198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.343199    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:16:45.343499 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.343215    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:16:45.343499 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.343233    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:16:45.343499 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.343250    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:16:45.343499 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.343263    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4h8h\" (UniqueName: \"kubernetes.io/projected/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7-kube-api-access-d4h8h\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:16:45.384727 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.384605    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"]
Oct 30 17:16:45.391215 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.391180    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-zjhkp"]
Oct 30 17:16:45.696914 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:45.696868    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7" path="/var/lib/kubelet/pods/47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7/volumes"
Oct 30 17:16:47.020985 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:47.020947    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7"
Oct 30 17:16:47.033861 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:47.033832    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7"
Oct 30 17:16:47.334038 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:47.333953    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7"
Oct 30 17:16:48.588822 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:48.588784    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2"]
Oct 30 17:16:48.589879 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:48.589039    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2" podUID="b40014e8-60cb-4b74-abbd-efa2eee22ad5" containerName="maas-api" containerID="cri-o://1e8e7592d9e6bbc6f12d82803290ee0b4cae50e21badf1f1652dd662ba00880e" gracePeriod=30
Oct 30 17:16:49.060427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:49.057635    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2"
Oct 30 17:16:49.143525 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:49.140916    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b40014e8-60cb-4b74-abbd-efa2eee22ad5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b40014e8-60cb-4b74-abbd-efa2eee22ad5-metrics-certs\") pod \"b40014e8-60cb-4b74-abbd-efa2eee22ad5\" (UID: \"b40014e8-60cb-4b74-abbd-efa2eee22ad5\") "
Oct 30 17:16:49.143525 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:49.140988    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b40014e8-60cb-4b74-abbd-efa2eee22ad5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b40014e8-60cb-4b74-abbd-efa2eee22ad5-openshift-service-ca\") pod \"b40014e8-60cb-4b74-abbd-efa2eee22ad5\" (UID: \"b40014e8-60cb-4b74-abbd-efa2eee22ad5\") "
Oct 30 17:16:49.143525 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:49.141250    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b40014e8-60cb-4b74-abbd-efa2eee22ad5-kube-api-access-s5gqv\" (UniqueName: \"kubernetes.io/projected/b40014e8-60cb-4b74-abbd-efa2eee22ad5-kube-api-access-s5gqv\") pod \"b40014e8-60cb-4b74-abbd-efa2eee22ad5\" (UID: \"b40014e8-60cb-4b74-abbd-efa2eee22ad5\") "
Oct 30 17:16:49.143525 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:49.141330    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b40014e8-60cb-4b74-abbd-efa2eee22ad5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b40014e8-60cb-4b74-abbd-efa2eee22ad5-maas-api-tls\") pod \"b40014e8-60cb-4b74-abbd-efa2eee22ad5\" (UID: \"b40014e8-60cb-4b74-abbd-efa2eee22ad5\") "
Oct 30 17:16:49.143525 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:49.142912    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b40014e8-60cb-4b74-abbd-efa2eee22ad5-openshift-service-ca" pod "b40014e8-60cb-4b74-abbd-efa2eee22ad5" (UID: "b40014e8-60cb-4b74-abbd-efa2eee22ad5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:16:49.153448 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:49.152833    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b40014e8-60cb-4b74-abbd-efa2eee22ad5-metrics-certs" pod "b40014e8-60cb-4b74-abbd-efa2eee22ad5" (UID: "b40014e8-60cb-4b74-abbd-efa2eee22ad5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:16:49.156433 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:49.154507    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b40014e8-60cb-4b74-abbd-efa2eee22ad5-maas-api-tls" pod "b40014e8-60cb-4b74-abbd-efa2eee22ad5" (UID: "b40014e8-60cb-4b74-abbd-efa2eee22ad5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:16:49.167433 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:49.164971    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b40014e8-60cb-4b74-abbd-efa2eee22ad5-kube-api-access-s5gqv" pod "b40014e8-60cb-4b74-abbd-efa2eee22ad5" (UID: "b40014e8-60cb-4b74-abbd-efa2eee22ad5"). InnerVolumeSpecName "kube-api-access-s5gqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:16:49.250490 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:49.244733    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5gqv\" (UniqueName: \"kubernetes.io/projected/b40014e8-60cb-4b74-abbd-efa2eee22ad5-kube-api-access-s5gqv\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:16:49.250490 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:49.244775    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b40014e8-60cb-4b74-abbd-efa2eee22ad5-maas-api-tls\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:16:49.250490 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:49.244789    4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b40014e8-60cb-4b74-abbd-efa2eee22ad5-metrics-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:16:49.250490 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:49.244804    4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b40014e8-60cb-4b74-abbd-efa2eee22ad5-openshift-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:16:49.341928 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:49.341892    4433 generic.go:356] "Generic (PLEG): container finished" podID="b40014e8-60cb-4b74-abbd-efa2eee22ad5" containerID="1e8e7592d9e6bbc6f12d82803290ee0b4cae50e21badf1f1652dd662ba00880e" exitCode=0
Oct 30 17:16:49.342115 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:49.341941    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2"
Oct 30 17:16:49.342115 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:49.341978    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2" event={"ID":"b40014e8-60cb-4b74-abbd-efa2eee22ad5","Type":"ContainerDied","Data":"1e8e7592d9e6bbc6f12d82803290ee0b4cae50e21badf1f1652dd662ba00880e"}
Oct 30 17:16:49.342115 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:49.342047    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2" event={"ID":"b40014e8-60cb-4b74-abbd-efa2eee22ad5","Type":"ContainerDied","Data":"149d313d8d21ec0da74b3385786fd2b8caac5a25203315cdb5559b4c706829e6"}
Oct 30 17:16:49.342268 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:49.342118    4433 scope.go:122] "RemoveContainer" containerID="1e8e7592d9e6bbc6f12d82803290ee0b4cae50e21badf1f1652dd662ba00880e"
Oct 30 17:16:49.379101 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:49.379070    4433 scope.go:122] "RemoveContainer" containerID="1e8e7592d9e6bbc6f12d82803290ee0b4cae50e21badf1f1652dd662ba00880e"
Oct 30 17:16:49.379477 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:16:49.379453    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e8e7592d9e6bbc6f12d82803290ee0b4cae50e21badf1f1652dd662ba00880e\": container with ID starting with 1e8e7592d9e6bbc6f12d82803290ee0b4cae50e21badf1f1652dd662ba00880e not found: ID does not exist" containerID="1e8e7592d9e6bbc6f12d82803290ee0b4cae50e21badf1f1652dd662ba00880e"
Oct 30 17:16:49.379574 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:49.379493    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e8e7592d9e6bbc6f12d82803290ee0b4cae50e21badf1f1652dd662ba00880e"} err="failed to get container status \"1e8e7592d9e6bbc6f12d82803290ee0b4cae50e21badf1f1652dd662ba00880e\": rpc error: code = NotFound desc = could not find container \"1e8e7592d9e6bbc6f12d82803290ee0b4cae50e21badf1f1652dd662ba00880e\": container with ID starting with 1e8e7592d9e6bbc6f12d82803290ee0b4cae50e21badf1f1652dd662ba00880e not found: ID does not exist"
Oct 30 17:16:49.450271 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:49.450235    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2"]
Oct 30 17:16:49.466721 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:49.466687    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39fecf-f85c8bb4b-kc6c2"]
Oct 30 17:16:49.697119 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:49.697081    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b40014e8-60cb-4b74-abbd-efa2eee22ad5" path="/var/lib/kubelet/pods/b40014e8-60cb-4b74-abbd-efa2eee22ad5/volumes"
Oct 30 17:16:53.773675 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:53.773641    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll"]
Oct 30 17:16:53.774263 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:53.774247    4433 cpu_manager.go:414] "containerMap: removing container" podUID="47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7" containerName="istio-proxy"
Oct 30 17:16:53.774306 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:53.774264    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7" containerName="istio-proxy"
Oct 30 17:16:53.774306 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:53.774282    4433 cpu_manager.go:414] "containerMap: removing container" podUID="b40014e8-60cb-4b74-abbd-efa2eee22ad5" containerName="maas-api"
Oct 30 17:16:53.774306 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:53.774287    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b40014e8-60cb-4b74-abbd-efa2eee22ad5" containerName="maas-api"
Oct 30 17:16:53.774433 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:53.774375    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="b40014e8-60cb-4b74-abbd-efa2eee22ad5" containerName="maas-api"
Oct 30 17:16:53.774433 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:53.774392    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="47d9a5e0-0550-44c4-9e38-5fcf9ad1a8b7" containerName="istio-proxy"
Oct 30 17:16:53.779151 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:53.779130    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll"
Oct 30 17:16:53.811799 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:53.811758    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll"]
Oct 30 17:16:53.814394 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:53.813377    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-e5ac8f-serving-cert\""
Oct 30 17:16:53.942607 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:53.942572    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95172c31-1b94-47fd-b95e-8170b67562ca-metrics-certs\") pod \"maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll\" (UID: \"95172c31-1b94-47fd-b95e-8170b67562ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll"
Oct 30 17:16:53.942765 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:53.942620    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95172c31-1b94-47fd-b95e-8170b67562ca-maas-api-tls\") pod \"maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll\" (UID: \"95172c31-1b94-47fd-b95e-8170b67562ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll"
Oct 30 17:16:53.942876 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:53.942856    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95172c31-1b94-47fd-b95e-8170b67562ca-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll\" (UID: \"95172c31-1b94-47fd-b95e-8170b67562ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll"
Oct 30 17:16:53.942919 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:53.942892    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfkwh\" (UniqueName: \"kubernetes.io/projected/95172c31-1b94-47fd-b95e-8170b67562ca-kube-api-access-dfkwh\") pod \"maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll\" (UID: \"95172c31-1b94-47fd-b95e-8170b67562ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll"
Oct 30 17:16:54.045100 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:54.045002    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95172c31-1b94-47fd-b95e-8170b67562ca-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll\" (UID: \"95172c31-1b94-47fd-b95e-8170b67562ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll"
Oct 30 17:16:54.045100 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:54.045068    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfkwh\" (UniqueName: \"kubernetes.io/projected/95172c31-1b94-47fd-b95e-8170b67562ca-kube-api-access-dfkwh\") pod \"maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll\" (UID: \"95172c31-1b94-47fd-b95e-8170b67562ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll"
Oct 30 17:16:54.045362 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:54.045184    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95172c31-1b94-47fd-b95e-8170b67562ca-metrics-certs\") pod \"maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll\" (UID: \"95172c31-1b94-47fd-b95e-8170b67562ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll"
Oct 30 17:16:54.045362 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:54.045226    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95172c31-1b94-47fd-b95e-8170b67562ca-maas-api-tls\") pod \"maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll\" (UID: \"95172c31-1b94-47fd-b95e-8170b67562ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll"
Oct 30 17:16:54.047245 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:54.046023    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95172c31-1b94-47fd-b95e-8170b67562ca-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll\" (UID: \"95172c31-1b94-47fd-b95e-8170b67562ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll"
Oct 30 17:16:54.052442 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:54.049694    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95172c31-1b94-47fd-b95e-8170b67562ca-metrics-certs\") pod \"maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll\" (UID: \"95172c31-1b94-47fd-b95e-8170b67562ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll"
Oct 30 17:16:54.059473 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:54.059006    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95172c31-1b94-47fd-b95e-8170b67562ca-maas-api-tls\") pod \"maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll\" (UID: \"95172c31-1b94-47fd-b95e-8170b67562ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll"
Oct 30 17:16:54.074185 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:54.074151    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfkwh\" (UniqueName: \"kubernetes.io/projected/95172c31-1b94-47fd-b95e-8170b67562ca-kube-api-access-dfkwh\") pod \"maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll\" (UID: \"95172c31-1b94-47fd-b95e-8170b67562ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll"
Oct 30 17:16:54.093982 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:54.093946    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll"
Oct 30 17:16:54.401609 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:54.401582    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll"]
Oct 30 17:16:54.404064 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:16:54.404032    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95172c31_1b94_47fd_b95e_8170b67562ca.slice/crio-db56297eedf36c22f69c3bd6a1fdb08ced8e421b61624033e08b928d12996432 WatchSource:0}: Error finding container db56297eedf36c22f69c3bd6a1fdb08ced8e421b61624033e08b928d12996432: Status 404 returned error can't find the container with id db56297eedf36c22f69c3bd6a1fdb08ced8e421b61624033e08b928d12996432
Oct 30 17:16:54.405658 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:54.405638    4433 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 17:16:55.385647 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:55.385615    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll" event={"ID":"95172c31-1b94-47fd-b95e-8170b67562ca","Type":"ContainerStarted","Data":"173af07945d3b74f12e8981b620b10898c1f9e284ff0273d920d20caf4fd8d46"}
Oct 30 17:16:55.386050 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:55.385664    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll" event={"ID":"95172c31-1b94-47fd-b95e-8170b67562ca","Type":"ContainerStarted","Data":"db56297eedf36c22f69c3bd6a1fdb08ced8e421b61624033e08b928d12996432"}
Oct 30 17:16:55.386050 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:55.385721    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll"
Oct 30 17:16:55.430142 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:55.430091    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll" podStartSLOduration=2.149915537 podStartE2EDuration="2.430076926s" totalImagesPullingTime="280.161389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:55.426724856 +0000 UTC m=+1791.861627062" watchObservedRunningTime="2026-10-30 17:16:55.430076926 +0000 UTC m=+1791.864979131"
Oct 30 17:16:56.978800 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:16:56.978759    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:16:58.373141 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:58.373104    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"]
Oct 30 17:16:58.373667 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:58.373444    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt" podUID="2cb53e81-fab8-43ff-98c2-e2f91d466226" containerName="istio-proxy" containerID="cri-o://a0166291b4b1ae17764b9fc4188ecd42a9b84698a59c48564284608e8c8e54ff" gracePeriod=30
Oct 30 17:16:59.701366 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:59.701331    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"]
Oct 30 17:16:59.706685 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:59.706662    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:16:59.727665 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:59.727640    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"]
Oct 30 17:16:59.746730 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:59.746697    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-987dfc-openshift-default-dockercfg-ggt87\""
Oct 30 17:16:59.747864 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:59.747829    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13bdf6f8-1211-4221-aa10-480d161dd3f8-istio-envoy\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:16:59.747995 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:59.747916    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz5mr\" (UniqueName: \"kubernetes.io/projected/13bdf6f8-1211-4221-aa10-480d161dd3f8-kube-api-access-hz5mr\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:16:59.748061 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:59.747995    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13bdf6f8-1211-4221-aa10-480d161dd3f8-credential-socket\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:16:59.748061 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:59.748042    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13bdf6f8-1211-4221-aa10-480d161dd3f8-istio-data\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:16:59.748171 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:59.748096    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13bdf6f8-1211-4221-aa10-480d161dd3f8-workload-socket\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:16:59.748171 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:59.748142    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13bdf6f8-1211-4221-aa10-480d161dd3f8-istio-token\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:16:59.748276 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:59.748191    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13bdf6f8-1211-4221-aa10-480d161dd3f8-istio-podinfo\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:16:59.748276 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:59.748262    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13bdf6f8-1211-4221-aa10-480d161dd3f8-workload-certs\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:16:59.748379 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:59.748307    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13bdf6f8-1211-4221-aa10-480d161dd3f8-istiod-ca-cert\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:16:59.849759 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:59.849708    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13bdf6f8-1211-4221-aa10-480d161dd3f8-workload-socket\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:16:59.849940 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:59.849785    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13bdf6f8-1211-4221-aa10-480d161dd3f8-istio-token\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:16:59.849940 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:59.849843    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13bdf6f8-1211-4221-aa10-480d161dd3f8-istio-podinfo\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:16:59.849940 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:59.849926    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13bdf6f8-1211-4221-aa10-480d161dd3f8-workload-certs\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:16:59.850130 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:59.849974    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13bdf6f8-1211-4221-aa10-480d161dd3f8-istiod-ca-cert\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:16:59.850130 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:59.850083    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13bdf6f8-1211-4221-aa10-480d161dd3f8-istio-envoy\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:16:59.850242 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:59.850163    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13bdf6f8-1211-4221-aa10-480d161dd3f8-workload-socket\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:16:59.850465 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:59.850436    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13bdf6f8-1211-4221-aa10-480d161dd3f8-workload-certs\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:16:59.850598 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:59.850478    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz5mr\" (UniqueName: \"kubernetes.io/projected/13bdf6f8-1211-4221-aa10-480d161dd3f8-kube-api-access-hz5mr\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:16:59.850598 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:59.850582    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13bdf6f8-1211-4221-aa10-480d161dd3f8-credential-socket\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:16:59.850689 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:59.850635    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13bdf6f8-1211-4221-aa10-480d161dd3f8-istio-data\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:16:59.850967 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:59.850944    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13bdf6f8-1211-4221-aa10-480d161dd3f8-istiod-ca-cert\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:16:59.851020 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:59.850972    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13bdf6f8-1211-4221-aa10-480d161dd3f8-istio-data\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:16:59.851221 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:59.851198    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13bdf6f8-1211-4221-aa10-480d161dd3f8-credential-socket\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:16:59.853372 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:59.853341    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13bdf6f8-1211-4221-aa10-480d161dd3f8-istio-envoy\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:16:59.853562 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:59.853539    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13bdf6f8-1211-4221-aa10-480d161dd3f8-istio-podinfo\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:16:59.873750 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:59.873719    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13bdf6f8-1211-4221-aa10-480d161dd3f8-istio-token\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:16:59.877138 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:16:59.877106    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz5mr\" (UniqueName: \"kubernetes.io/projected/13bdf6f8-1211-4221-aa10-480d161dd3f8-kube-api-access-hz5mr\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:17:00.039480 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:00.029704    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:17:00.468982 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:00.468952    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"]
Oct 30 17:17:00.471838 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:17:00.471807    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13bdf6f8_1211_4221_aa10_480d161dd3f8.slice/crio-c958b2f4800da464ec41a65eafd07c9a555b090ebd7f2462839f39f2cb1c6fef WatchSource:0}: Error finding container c958b2f4800da464ec41a65eafd07c9a555b090ebd7f2462839f39f2cb1c6fef: Status 404 returned error can't find the container with id c958b2f4800da464ec41a65eafd07c9a555b090ebd7f2462839f39f2cb1c6fef
Oct 30 17:17:00.474984 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:00.474944    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:17:00.475057 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:00.475037    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:17:00.475099 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:00.475082    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:17:01.397857 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:01.397832    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll"
Oct 30 17:17:01.425649 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:01.425616    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh" event={"ID":"13bdf6f8-1211-4221-aa10-480d161dd3f8","Type":"ContainerStarted","Data":"c11e45dd799b0879162a088605376060ff0d44770d15d155b568365d3a6d7fb7"}
Oct 30 17:17:01.425830 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:01.425669    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh" event={"ID":"13bdf6f8-1211-4221-aa10-480d161dd3f8","Type":"ContainerStarted","Data":"c958b2f4800da464ec41a65eafd07c9a555b090ebd7f2462839f39f2cb1c6fef"}
Oct 30 17:17:01.492464 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:01.492386    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh" podStartSLOduration=2.4923663449999998 podStartE2EDuration="2.492366345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:17:01.488481254 +0000 UTC m=+1797.923383460" watchObservedRunningTime="2026-10-30 17:17:01.492366345 +0000 UTC m=+1797.927268553"
Oct 30 17:17:02.030644 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:02.030609    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:17:02.036250 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:02.036226    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:17:02.431785 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:02.431750    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:17:02.432859 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:02.432840    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:17:03.659251 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:03.659185    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:17:03.739427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:03.723505    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb53e81-fab8-43ff-98c2-e2f91d466226-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb53e81-fab8-43ff-98c2-e2f91d466226-workload-socket\") pod \"2cb53e81-fab8-43ff-98c2-e2f91d466226\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") "
Oct 30 17:17:03.739427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:03.730896    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb53e81-fab8-43ff-98c2-e2f91d466226-workload-socket" pod "2cb53e81-fab8-43ff-98c2-e2f91d466226" (UID: "2cb53e81-fab8-43ff-98c2-e2f91d466226"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:17:03.824844 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:03.824806    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cb53e81-fab8-43ff-98c2-e2f91d466226-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cb53e81-fab8-43ff-98c2-e2f91d466226-istiod-ca-cert\") pod \"2cb53e81-fab8-43ff-98c2-e2f91d466226\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") "
Oct 30 17:17:03.825027 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:03.824867    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb53e81-fab8-43ff-98c2-e2f91d466226-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb53e81-fab8-43ff-98c2-e2f91d466226-credential-socket\") pod \"2cb53e81-fab8-43ff-98c2-e2f91d466226\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") "
Oct 30 17:17:03.825027 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:03.824944    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb53e81-fab8-43ff-98c2-e2f91d466226-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cb53e81-fab8-43ff-98c2-e2f91d466226-istio-envoy\") pod \"2cb53e81-fab8-43ff-98c2-e2f91d466226\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") "
Oct 30 17:17:03.825128 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:03.825106    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2cb53e81-fab8-43ff-98c2-e2f91d466226-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cb53e81-fab8-43ff-98c2-e2f91d466226-istio-podinfo\") pod \"2cb53e81-fab8-43ff-98c2-e2f91d466226\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") "
Oct 30 17:17:03.825210 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:03.825189    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb53e81-fab8-43ff-98c2-e2f91d466226-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cb53e81-fab8-43ff-98c2-e2f91d466226-istio-data\") pod \"2cb53e81-fab8-43ff-98c2-e2f91d466226\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") "
Oct 30 17:17:03.825297 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:03.825228    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cb53e81-fab8-43ff-98c2-e2f91d466226-istiod-ca-cert" pod "2cb53e81-fab8-43ff-98c2-e2f91d466226" (UID: "2cb53e81-fab8-43ff-98c2-e2f91d466226"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:17:03.825297 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:03.825229    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb53e81-fab8-43ff-98c2-e2f91d466226-credential-socket" pod "2cb53e81-fab8-43ff-98c2-e2f91d466226" (UID: "2cb53e81-fab8-43ff-98c2-e2f91d466226"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:17:03.825297 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:03.825292    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cb53e81-fab8-43ff-98c2-e2f91d466226-kube-api-access-rw9xx\" (UniqueName: \"kubernetes.io/projected/2cb53e81-fab8-43ff-98c2-e2f91d466226-kube-api-access-rw9xx\") pod \"2cb53e81-fab8-43ff-98c2-e2f91d466226\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") "
Oct 30 17:17:03.825914 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:03.825340    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb53e81-fab8-43ff-98c2-e2f91d466226-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cb53e81-fab8-43ff-98c2-e2f91d466226-workload-certs\") pod \"2cb53e81-fab8-43ff-98c2-e2f91d466226\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") "
Oct 30 17:17:03.825914 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:03.825425    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cb53e81-fab8-43ff-98c2-e2f91d466226-istio-token\" (UniqueName: \"kubernetes.io/projected/2cb53e81-fab8-43ff-98c2-e2f91d466226-istio-token\") pod \"2cb53e81-fab8-43ff-98c2-e2f91d466226\" (UID: \"2cb53e81-fab8-43ff-98c2-e2f91d466226\") "
Oct 30 17:17:03.825914 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:03.825538    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb53e81-fab8-43ff-98c2-e2f91d466226-istio-data" pod "2cb53e81-fab8-43ff-98c2-e2f91d466226" (UID: "2cb53e81-fab8-43ff-98c2-e2f91d466226"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:17:03.825914 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:03.825699    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb53e81-fab8-43ff-98c2-e2f91d466226-workload-certs" pod "2cb53e81-fab8-43ff-98c2-e2f91d466226" (UID: "2cb53e81-fab8-43ff-98c2-e2f91d466226"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:17:03.826097 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:03.826019    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cb53e81-fab8-43ff-98c2-e2f91d466226-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:17:03.826097 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:03.826043    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cb53e81-fab8-43ff-98c2-e2f91d466226-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:17:03.826097 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:03.826057    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb53e81-fab8-43ff-98c2-e2f91d466226-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:17:03.826097 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:03.826069    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cb53e81-fab8-43ff-98c2-e2f91d466226-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:17:03.826097 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:03.826082    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb53e81-fab8-43ff-98c2-e2f91d466226-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:17:03.830784 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:03.830757    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb53e81-fab8-43ff-98c2-e2f91d466226-istio-envoy" pod "2cb53e81-fab8-43ff-98c2-e2f91d466226" (UID: "2cb53e81-fab8-43ff-98c2-e2f91d466226"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:17:03.830907 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:03.830804    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb53e81-fab8-43ff-98c2-e2f91d466226-istio-token" pod "2cb53e81-fab8-43ff-98c2-e2f91d466226" (UID: "2cb53e81-fab8-43ff-98c2-e2f91d466226"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:17:03.835822 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:03.835788    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2cb53e81-fab8-43ff-98c2-e2f91d466226-istio-podinfo" pod "2cb53e81-fab8-43ff-98c2-e2f91d466226" (UID: "2cb53e81-fab8-43ff-98c2-e2f91d466226"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:17:03.835952 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:03.835882    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb53e81-fab8-43ff-98c2-e2f91d466226-kube-api-access-rw9xx" pod "2cb53e81-fab8-43ff-98c2-e2f91d466226" (UID: "2cb53e81-fab8-43ff-98c2-e2f91d466226"). InnerVolumeSpecName "kube-api-access-rw9xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:17:03.926985 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:03.926956    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw9xx\" (UniqueName: \"kubernetes.io/projected/2cb53e81-fab8-43ff-98c2-e2f91d466226-kube-api-access-rw9xx\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:17:03.926985 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:03.926983    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cb53e81-fab8-43ff-98c2-e2f91d466226-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:17:03.927164 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:03.926994    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cb53e81-fab8-43ff-98c2-e2f91d466226-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:17:03.927164 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:03.927003    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cb53e81-fab8-43ff-98c2-e2f91d466226-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:17:04.452326 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:04.451580    4433 generic.go:356] "Generic (PLEG): container finished" podID="2cb53e81-fab8-43ff-98c2-e2f91d466226" containerID="a0166291b4b1ae17764b9fc4188ecd42a9b84698a59c48564284608e8c8e54ff" exitCode=0
Oct 30 17:17:04.452326 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:04.451627    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt" event={"ID":"2cb53e81-fab8-43ff-98c2-e2f91d466226","Type":"ContainerDied","Data":"a0166291b4b1ae17764b9fc4188ecd42a9b84698a59c48564284608e8c8e54ff"}
Oct 30 17:17:04.452326 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:04.451699    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt" event={"ID":"2cb53e81-fab8-43ff-98c2-e2f91d466226","Type":"ContainerDied","Data":"62bdc9d34bfdb63e6f9bad12ce88794f6e5f531c552376056e10166824d1e298"}
Oct 30 17:17:04.452326 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:04.451729    4433 scope.go:122] "RemoveContainer" containerID="a0166291b4b1ae17764b9fc4188ecd42a9b84698a59c48564284608e8c8e54ff"
Oct 30 17:17:04.452789 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:04.452385    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"
Oct 30 17:17:04.487052 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:04.487024    4433 scope.go:122] "RemoveContainer" containerID="6d4602c51365ec90f03c961d88ff94cb0622649f4408d7387becadfe805181f4"
Oct 30 17:17:04.497182 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:04.497037    4433 scope.go:122] "RemoveContainer" containerID="a0166291b4b1ae17764b9fc4188ecd42a9b84698a59c48564284608e8c8e54ff"
Oct 30 17:17:04.497358 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:17:04.497337    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0166291b4b1ae17764b9fc4188ecd42a9b84698a59c48564284608e8c8e54ff\": container with ID starting with a0166291b4b1ae17764b9fc4188ecd42a9b84698a59c48564284608e8c8e54ff not found: ID does not exist" containerID="a0166291b4b1ae17764b9fc4188ecd42a9b84698a59c48564284608e8c8e54ff"
Oct 30 17:17:04.497437 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:04.497365    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0166291b4b1ae17764b9fc4188ecd42a9b84698a59c48564284608e8c8e54ff"} err="failed to get container status \"a0166291b4b1ae17764b9fc4188ecd42a9b84698a59c48564284608e8c8e54ff\": rpc error: code = NotFound desc = could not find container \"a0166291b4b1ae17764b9fc4188ecd42a9b84698a59c48564284608e8c8e54ff\": container with ID starting with a0166291b4b1ae17764b9fc4188ecd42a9b84698a59c48564284608e8c8e54ff not found: ID does not exist"
Oct 30 17:17:04.497437 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:04.497385    4433 scope.go:122] "RemoveContainer" containerID="6d4602c51365ec90f03c961d88ff94cb0622649f4408d7387becadfe805181f4"
Oct 30 17:17:04.497720 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:17:04.497697    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d4602c51365ec90f03c961d88ff94cb0622649f4408d7387becadfe805181f4\": container with ID starting with 6d4602c51365ec90f03c961d88ff94cb0622649f4408d7387becadfe805181f4 not found: ID does not exist" containerID="6d4602c51365ec90f03c961d88ff94cb0622649f4408d7387becadfe805181f4"
Oct 30 17:17:04.497778 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:04.497729    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d4602c51365ec90f03c961d88ff94cb0622649f4408d7387becadfe805181f4"} err="failed to get container status \"6d4602c51365ec90f03c961d88ff94cb0622649f4408d7387becadfe805181f4\": rpc error: code = NotFound desc = could not find container \"6d4602c51365ec90f03c961d88ff94cb0622649f4408d7387becadfe805181f4\": container with ID starting with 6d4602c51365ec90f03c961d88ff94cb0622649f4408d7387becadfe805181f4 not found: ID does not exist"
Oct 30 17:17:04.523174 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:04.523083    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"]
Oct 30 17:17:04.527225 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:04.527200    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-6rkbt"]
Oct 30 17:17:05.697757 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:05.697725    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cb53e81-fab8-43ff-98c2-e2f91d466226" path="/var/lib/kubelet/pods/2cb53e81-fab8-43ff-98c2-e2f91d466226/volumes"
Oct 30 17:17:07.441598 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:07.441565    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55"]
Oct 30 17:17:07.442589 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:07.442564    4433 cpu_manager.go:414] "containerMap: removing container" podUID="2cb53e81-fab8-43ff-98c2-e2f91d466226" containerName="istio-proxy"
Oct 30 17:17:07.442734 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:07.442598    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb53e81-fab8-43ff-98c2-e2f91d466226" containerName="istio-proxy"
Oct 30 17:17:07.442734 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:07.442617    4433 cpu_manager.go:414] "containerMap: removing container" podUID="2cb53e81-fab8-43ff-98c2-e2f91d466226" containerName="istio-proxy"
Oct 30 17:17:07.442734 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:07.442627    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb53e81-fab8-43ff-98c2-e2f91d466226" containerName="istio-proxy"
Oct 30 17:17:07.442890 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:07.442774    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cb53e81-fab8-43ff-98c2-e2f91d466226" containerName="istio-proxy"
Oct 30 17:17:07.442890 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:07.442787    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cb53e81-fab8-43ff-98c2-e2f91d466226" containerName="istio-proxy"
Oct 30 17:17:07.448173 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:07.448148    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55"
Oct 30 17:17:07.466323 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:07.466292    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55"]
Oct 30 17:17:07.473079 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:07.473053    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c82666\"/\"openshift-service-ca.crt\""
Oct 30 17:17:07.473079 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:07.473053    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c82666\"/\"kube-root-ca.crt\""
Oct 30 17:17:07.473296 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:07.473130    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c82666\"/\"default-dockercfg-7vzwl\""
Oct 30 17:17:07.476161 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:07.476138    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c82666\"/\"test-model-c82666-kserve-self-signed-certs\""
Oct 30 17:17:07.569558 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:07.569520    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzps4\" (UniqueName: \"kubernetes.io/projected/8f245c01-8039-4ce3-a4de-ce744f348adc-kube-api-access-tzps4\") pod \"test-model-c82666-kserve-686458489c-fvn55\" (UID: \"8f245c01-8039-4ce3-a4de-ce744f348adc\") " pod="ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55"
Oct 30 17:17:07.569743 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:07.569580    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f245c01-8039-4ce3-a4de-ce744f348adc-tls-certs\") pod \"test-model-c82666-kserve-686458489c-fvn55\" (UID: \"8f245c01-8039-4ce3-a4de-ce744f348adc\") " pod="ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55"
Oct 30 17:17:07.569743 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:07.569688    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f245c01-8039-4ce3-a4de-ce744f348adc-home\") pod \"test-model-c82666-kserve-686458489c-fvn55\" (UID: \"8f245c01-8039-4ce3-a4de-ce744f348adc\") " pod="ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55"
Oct 30 17:17:07.569841 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:07.569744    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f245c01-8039-4ce3-a4de-ce744f348adc-tmp-dir\") pod \"test-model-c82666-kserve-686458489c-fvn55\" (UID: \"8f245c01-8039-4ce3-a4de-ce744f348adc\") " pod="ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55"
Oct 30 17:17:07.569883 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:07.569852    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f245c01-8039-4ce3-a4de-ce744f348adc-model-cache\") pod \"test-model-c82666-kserve-686458489c-fvn55\" (UID: \"8f245c01-8039-4ce3-a4de-ce744f348adc\") " pod="ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55"
Oct 30 17:17:07.569933 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:07.569892    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f245c01-8039-4ce3-a4de-ce744f348adc-dshm\") pod \"test-model-c82666-kserve-686458489c-fvn55\" (UID: \"8f245c01-8039-4ce3-a4de-ce744f348adc\") " pod="ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55"
Oct 30 17:17:07.671757 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:07.671717    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f245c01-8039-4ce3-a4de-ce744f348adc-model-cache\") pod \"test-model-c82666-kserve-686458489c-fvn55\" (UID: \"8f245c01-8039-4ce3-a4de-ce744f348adc\") " pod="ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55"
Oct 30 17:17:07.671977 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:07.671787    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f245c01-8039-4ce3-a4de-ce744f348adc-dshm\") pod \"test-model-c82666-kserve-686458489c-fvn55\" (UID: \"8f245c01-8039-4ce3-a4de-ce744f348adc\") " pod="ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55"
Oct 30 17:17:07.672039 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:07.671995    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzps4\" (UniqueName: \"kubernetes.io/projected/8f245c01-8039-4ce3-a4de-ce744f348adc-kube-api-access-tzps4\") pod \"test-model-c82666-kserve-686458489c-fvn55\" (UID: \"8f245c01-8039-4ce3-a4de-ce744f348adc\") " pod="ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55"
Oct 30 17:17:07.672092 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:07.672045    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f245c01-8039-4ce3-a4de-ce744f348adc-tls-certs\") pod \"test-model-c82666-kserve-686458489c-fvn55\" (UID: \"8f245c01-8039-4ce3-a4de-ce744f348adc\") " pod="ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55"
Oct 30 17:17:07.672092 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:07.672084    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f245c01-8039-4ce3-a4de-ce744f348adc-home\") pod \"test-model-c82666-kserve-686458489c-fvn55\" (UID: \"8f245c01-8039-4ce3-a4de-ce744f348adc\") " pod="ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55"
Oct 30 17:17:07.672222 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:07.672133    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f245c01-8039-4ce3-a4de-ce744f348adc-tmp-dir\") pod \"test-model-c82666-kserve-686458489c-fvn55\" (UID: \"8f245c01-8039-4ce3-a4de-ce744f348adc\") " pod="ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55"
Oct 30 17:17:07.672222 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:07.672179    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f245c01-8039-4ce3-a4de-ce744f348adc-model-cache\") pod \"test-model-c82666-kserve-686458489c-fvn55\" (UID: \"8f245c01-8039-4ce3-a4de-ce744f348adc\") " pod="ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55"
Oct 30 17:17:07.672524 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:07.672490    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f245c01-8039-4ce3-a4de-ce744f348adc-tmp-dir\") pod \"test-model-c82666-kserve-686458489c-fvn55\" (UID: \"8f245c01-8039-4ce3-a4de-ce744f348adc\") " pod="ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55"
Oct 30 17:17:07.672684 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:07.672656    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f245c01-8039-4ce3-a4de-ce744f348adc-home\") pod \"test-model-c82666-kserve-686458489c-fvn55\" (UID: \"8f245c01-8039-4ce3-a4de-ce744f348adc\") " pod="ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55"
Oct 30 17:17:07.674725 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:07.674700    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f245c01-8039-4ce3-a4de-ce744f348adc-dshm\") pod \"test-model-c82666-kserve-686458489c-fvn55\" (UID: \"8f245c01-8039-4ce3-a4de-ce744f348adc\") " pod="ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55"
Oct 30 17:17:07.674894 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:07.674877    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f245c01-8039-4ce3-a4de-ce744f348adc-tls-certs\") pod \"test-model-c82666-kserve-686458489c-fvn55\" (UID: \"8f245c01-8039-4ce3-a4de-ce744f348adc\") " pod="ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55"
Oct 30 17:17:07.699939 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:07.699862    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzps4\" (UniqueName: \"kubernetes.io/projected/8f245c01-8039-4ce3-a4de-ce744f348adc-kube-api-access-tzps4\") pod \"test-model-c82666-kserve-686458489c-fvn55\" (UID: \"8f245c01-8039-4ce3-a4de-ce744f348adc\") " pod="ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55"
Oct 30 17:17:07.772424 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:07.771685    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55"
Oct 30 17:17:08.225273 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:08.225159    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55"]
Oct 30 17:17:08.229911 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:17:08.229875    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f245c01_8039_4ce3_a4de_ce744f348adc.slice/crio-35fa446cc05779abbe28aa19427c08a338d4df51a1ea96dc51407f2738248619 WatchSource:0}: Error finding container 35fa446cc05779abbe28aa19427c08a338d4df51a1ea96dc51407f2738248619: Status 404 returned error can't find the container with id 35fa446cc05779abbe28aa19427c08a338d4df51a1ea96dc51407f2738248619
Oct 30 17:17:08.548717 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:08.548529    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55" event={"ID":"8f245c01-8039-4ce3-a4de-ce744f348adc","Type":"ContainerStarted","Data":"5420b797c0a27f6734aea206a4d484be716351c99bf947d07b7383fc33e60aa3"}
Oct 30 17:17:08.548717 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:08.548645    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55" event={"ID":"8f245c01-8039-4ce3-a4de-ce744f348adc","Type":"ContainerStarted","Data":"35fa446cc05779abbe28aa19427c08a338d4df51a1ea96dc51407f2738248619"}
Oct 30 17:17:08.590110 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:08.590016    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55" podStartSLOduration=1.590003294 podStartE2EDuration="1.590003294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:17:08.589212255 +0000 UTC m=+1805.024114465" watchObservedRunningTime="2026-10-30 17:17:08.590003294 +0000 UTC m=+1805.024905499"
Oct 30 17:17:08.772442 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:08.772393    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55"
Oct 30 17:17:14.292660 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:14.292623    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg"]
Oct 30 17:17:14.297384 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:14.297353    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg"
Oct 30 17:17:14.316988 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:14.316958    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-dd5c8f-serving-cert\""
Oct 30 17:17:14.322951 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:14.322431    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg"]
Oct 30 17:17:14.395285 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:14.395245    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a8f95b8-4923-4d7c-8b6b-2e65231eeb09-metrics-certs\") pod \"maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg\" (UID: \"7a8f95b8-4923-4d7c-8b6b-2e65231eeb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg"
Oct 30 17:17:14.395451 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:14.395315    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj8lp\" (UniqueName: \"kubernetes.io/projected/7a8f95b8-4923-4d7c-8b6b-2e65231eeb09-kube-api-access-gj8lp\") pod \"maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg\" (UID: \"7a8f95b8-4923-4d7c-8b6b-2e65231eeb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg"
Oct 30 17:17:14.395512 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:14.395442    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a8f95b8-4923-4d7c-8b6b-2e65231eeb09-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg\" (UID: \"7a8f95b8-4923-4d7c-8b6b-2e65231eeb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg"
Oct 30 17:17:14.395546 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:14.395526    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a8f95b8-4923-4d7c-8b6b-2e65231eeb09-maas-api-tls\") pod \"maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg\" (UID: \"7a8f95b8-4923-4d7c-8b6b-2e65231eeb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg"
Oct 30 17:17:14.503160 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:14.501905    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a8f95b8-4923-4d7c-8b6b-2e65231eeb09-metrics-certs\") pod \"maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg\" (UID: \"7a8f95b8-4923-4d7c-8b6b-2e65231eeb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg"
Oct 30 17:17:14.503160 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:14.502010    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj8lp\" (UniqueName: \"kubernetes.io/projected/7a8f95b8-4923-4d7c-8b6b-2e65231eeb09-kube-api-access-gj8lp\") pod \"maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg\" (UID: \"7a8f95b8-4923-4d7c-8b6b-2e65231eeb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg"
Oct 30 17:17:14.503160 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:14.502065    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a8f95b8-4923-4d7c-8b6b-2e65231eeb09-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg\" (UID: \"7a8f95b8-4923-4d7c-8b6b-2e65231eeb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg"
Oct 30 17:17:14.503160 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:14.502105    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a8f95b8-4923-4d7c-8b6b-2e65231eeb09-maas-api-tls\") pod \"maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg\" (UID: \"7a8f95b8-4923-4d7c-8b6b-2e65231eeb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg"
Oct 30 17:17:14.504432 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:14.504365    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a8f95b8-4923-4d7c-8b6b-2e65231eeb09-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg\" (UID: \"7a8f95b8-4923-4d7c-8b6b-2e65231eeb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg"
Oct 30 17:17:14.515112 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:14.514845    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a8f95b8-4923-4d7c-8b6b-2e65231eeb09-maas-api-tls\") pod \"maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg\" (UID: \"7a8f95b8-4923-4d7c-8b6b-2e65231eeb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg"
Oct 30 17:17:14.515730 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:14.515700    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a8f95b8-4923-4d7c-8b6b-2e65231eeb09-metrics-certs\") pod \"maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg\" (UID: \"7a8f95b8-4923-4d7c-8b6b-2e65231eeb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg"
Oct 30 17:17:14.530266 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:14.530219    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj8lp\" (UniqueName: \"kubernetes.io/projected/7a8f95b8-4923-4d7c-8b6b-2e65231eeb09-kube-api-access-gj8lp\") pod \"maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg\" (UID: \"7a8f95b8-4923-4d7c-8b6b-2e65231eeb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg"
Oct 30 17:17:14.616742 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:14.616641    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg"
Oct 30 17:17:14.926509 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:14.926475    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg"]
Oct 30 17:17:14.927769 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:17:14.927741    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a8f95b8_4923_4d7c_8b6b_2e65231eeb09.slice/crio-1e4c2da0aadf3ebd42698bebc799d08a018df81c1dabf47fba39d04e2486cb1e WatchSource:0}: Error finding container 1e4c2da0aadf3ebd42698bebc799d08a018df81c1dabf47fba39d04e2486cb1e: Status 404 returned error can't find the container with id 1e4c2da0aadf3ebd42698bebc799d08a018df81c1dabf47fba39d04e2486cb1e
Oct 30 17:17:15.600239 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:15.600146    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg" event={"ID":"7a8f95b8-4923-4d7c-8b6b-2e65231eeb09","Type":"ContainerStarted","Data":"e1a2d5fa9d4ed90cd63091c2573e57241dda07d00eac28634dce726330efa8dc"}
Oct 30 17:17:15.600239 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:15.600207    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg" event={"ID":"7a8f95b8-4923-4d7c-8b6b-2e65231eeb09","Type":"ContainerStarted","Data":"1e4c2da0aadf3ebd42698bebc799d08a018df81c1dabf47fba39d04e2486cb1e"}
Oct 30 17:17:15.600704 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:15.600264    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg"
Oct 30 17:17:15.707975 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:15.706651    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg" podStartSLOduration=1.314670147 podStartE2EDuration="1.706632419s" totalImagesPullingTime="391.962272ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:17:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:17:15.659483394 +0000 UTC m=+1812.094385601" watchObservedRunningTime="2026-10-30 17:17:15.706632419 +0000 UTC m=+1812.141534626"
Oct 30 17:17:15.835284 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:15.835236    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc"]
Oct 30 17:17:15.861517 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:15.842677    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc"
Oct 30 17:17:15.861517 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:15.857991    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5d67ca\"/\"test-model-5d67ca-kserve-self-signed-certs\""
Oct 30 17:17:15.861517 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:15.858355    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5d67ca\"/\"default-dockercfg-28f8v\""
Oct 30 17:17:15.861517 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:15.858451    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5d67ca\"/\"openshift-service-ca.crt\""
Oct 30 17:17:15.861517 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:15.858545    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5d67ca\"/\"kube-root-ca.crt\""
Oct 30 17:17:15.865046 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:15.865016    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc"]
Oct 30 17:17:15.928317 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:15.928275    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36e810bc-53a4-46a7-9ccc-805373681902-model-cache\") pod \"test-model-5d67ca-kserve-7595bf8795-m77qc\" (UID: \"36e810bc-53a4-46a7-9ccc-805373681902\") " pod="ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc"
Oct 30 17:17:15.928582 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:15.928554    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36e810bc-53a4-46a7-9ccc-805373681902-tls-certs\") pod \"test-model-5d67ca-kserve-7595bf8795-m77qc\" (UID: \"36e810bc-53a4-46a7-9ccc-805373681902\") " pod="ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc"
Oct 30 17:17:15.928656 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:15.928606    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36e810bc-53a4-46a7-9ccc-805373681902-dshm\") pod \"test-model-5d67ca-kserve-7595bf8795-m77qc\" (UID: \"36e810bc-53a4-46a7-9ccc-805373681902\") " pod="ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc"
Oct 30 17:17:15.928711 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:15.928670    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36e810bc-53a4-46a7-9ccc-805373681902-home\") pod \"test-model-5d67ca-kserve-7595bf8795-m77qc\" (UID: \"36e810bc-53a4-46a7-9ccc-805373681902\") " pod="ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc"
Oct 30 17:17:15.928711 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:15.928700    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmnzx\" (UniqueName: \"kubernetes.io/projected/36e810bc-53a4-46a7-9ccc-805373681902-kube-api-access-cmnzx\") pod \"test-model-5d67ca-kserve-7595bf8795-m77qc\" (UID: \"36e810bc-53a4-46a7-9ccc-805373681902\") " pod="ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc"
Oct 30 17:17:15.928829 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:15.928820    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36e810bc-53a4-46a7-9ccc-805373681902-tmp-dir\") pod \"test-model-5d67ca-kserve-7595bf8795-m77qc\" (UID: \"36e810bc-53a4-46a7-9ccc-805373681902\") " pod="ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc"
Oct 30 17:17:16.030361 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:16.030316    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36e810bc-53a4-46a7-9ccc-805373681902-tmp-dir\") pod \"test-model-5d67ca-kserve-7595bf8795-m77qc\" (UID: \"36e810bc-53a4-46a7-9ccc-805373681902\") " pod="ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc"
Oct 30 17:17:16.030550 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:16.030470    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36e810bc-53a4-46a7-9ccc-805373681902-model-cache\") pod \"test-model-5d67ca-kserve-7595bf8795-m77qc\" (UID: \"36e810bc-53a4-46a7-9ccc-805373681902\") " pod="ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc"
Oct 30 17:17:16.030763 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:16.030742    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36e810bc-53a4-46a7-9ccc-805373681902-tls-certs\") pod \"test-model-5d67ca-kserve-7595bf8795-m77qc\" (UID: \"36e810bc-53a4-46a7-9ccc-805373681902\") " pod="ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc"
Oct 30 17:17:16.030866 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:16.030759    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36e810bc-53a4-46a7-9ccc-805373681902-tmp-dir\") pod \"test-model-5d67ca-kserve-7595bf8795-m77qc\" (UID: \"36e810bc-53a4-46a7-9ccc-805373681902\") " pod="ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc"
Oct 30 17:17:16.030866 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:16.030810    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36e810bc-53a4-46a7-9ccc-805373681902-dshm\") pod \"test-model-5d67ca-kserve-7595bf8795-m77qc\" (UID: \"36e810bc-53a4-46a7-9ccc-805373681902\") " pod="ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc"
Oct 30 17:17:16.030866 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:16.030836    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36e810bc-53a4-46a7-9ccc-805373681902-home\") pod \"test-model-5d67ca-kserve-7595bf8795-m77qc\" (UID: \"36e810bc-53a4-46a7-9ccc-805373681902\") " pod="ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc"
Oct 30 17:17:16.030866 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:16.030860    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmnzx\" (UniqueName: \"kubernetes.io/projected/36e810bc-53a4-46a7-9ccc-805373681902-kube-api-access-cmnzx\") pod \"test-model-5d67ca-kserve-7595bf8795-m77qc\" (UID: \"36e810bc-53a4-46a7-9ccc-805373681902\") " pod="ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc"
Oct 30 17:17:16.031082 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:16.030811    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36e810bc-53a4-46a7-9ccc-805373681902-model-cache\") pod \"test-model-5d67ca-kserve-7595bf8795-m77qc\" (UID: \"36e810bc-53a4-46a7-9ccc-805373681902\") " pod="ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc"
Oct 30 17:17:16.031375 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:16.031356    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36e810bc-53a4-46a7-9ccc-805373681902-home\") pod \"test-model-5d67ca-kserve-7595bf8795-m77qc\" (UID: \"36e810bc-53a4-46a7-9ccc-805373681902\") " pod="ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc"
Oct 30 17:17:16.033587 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:16.033560    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36e810bc-53a4-46a7-9ccc-805373681902-dshm\") pod \"test-model-5d67ca-kserve-7595bf8795-m77qc\" (UID: \"36e810bc-53a4-46a7-9ccc-805373681902\") " pod="ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc"
Oct 30 17:17:16.033744 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:16.033725    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36e810bc-53a4-46a7-9ccc-805373681902-tls-certs\") pod \"test-model-5d67ca-kserve-7595bf8795-m77qc\" (UID: \"36e810bc-53a4-46a7-9ccc-805373681902\") " pod="ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc"
Oct 30 17:17:16.053953 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:16.053925    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmnzx\" (UniqueName: \"kubernetes.io/projected/36e810bc-53a4-46a7-9ccc-805373681902-kube-api-access-cmnzx\") pod \"test-model-5d67ca-kserve-7595bf8795-m77qc\" (UID: \"36e810bc-53a4-46a7-9ccc-805373681902\") " pod="ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc"
Oct 30 17:17:16.169480 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:16.167856    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc"
Oct 30 17:17:16.579440 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:16.579383    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc"]
Oct 30 17:17:16.582551 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:17:16.582520    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36e810bc_53a4_46a7_9ccc_805373681902.slice/crio-aedeacd74925d0a78c7a25c5f632b57fce170a9bea9a712e3da8588ccb8ec4b4 WatchSource:0}: Error finding container aedeacd74925d0a78c7a25c5f632b57fce170a9bea9a712e3da8588ccb8ec4b4: Status 404 returned error can't find the container with id aedeacd74925d0a78c7a25c5f632b57fce170a9bea9a712e3da8588ccb8ec4b4
Oct 30 17:17:16.608192 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:16.608161    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc" event={"ID":"36e810bc-53a4-46a7-9ccc-805373681902","Type":"ContainerStarted","Data":"aedeacd74925d0a78c7a25c5f632b57fce170a9bea9a712e3da8588ccb8ec4b4"}
Oct 30 17:17:17.616244 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:17.616207    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc" event={"ID":"36e810bc-53a4-46a7-9ccc-805373681902","Type":"ContainerStarted","Data":"f838bbdb57b2b280ef873a79ab812ef5d2d1292b57666b9ab32931920b601499"}
Oct 30 17:17:17.658072 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:17.658022    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc" podStartSLOduration=2.6580083180000003 podStartE2EDuration="2.658008318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:17:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:17:17.657191471 +0000 UTC m=+1814.092093677" watchObservedRunningTime="2026-10-30 17:17:17.658008318 +0000 UTC m=+1814.092910523"
Oct 30 17:17:17.781746 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:17.781705    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55"
Oct 30 17:17:17.851626 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:17.851591    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55"
Oct 30 17:17:18.168493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:18.168460    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc"
Oct 30 17:17:18.636262 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:18.636226    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55"
Oct 30 17:17:21.613922 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:21.613891    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg"
Oct 30 17:17:25.912535 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:25.912500    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"]
Oct 30 17:17:25.913057 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:25.912718    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7" podUID="21dc53b6-74d2-48a6-b5dd-3b1d2940e845" containerName="istio-proxy" containerID="cri-o://244f946d18f2dc54e9e26f3fd19380cfc225c9862e2096f4fda2778eda487017" gracePeriod=30
Oct 30 17:17:26.168665 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:26.168567    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc"
Oct 30 17:17:26.181532 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:26.181502    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc"
Oct 30 17:17:26.691058 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:26.691022    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc"
Oct 30 17:17:26.978662 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:17:26.978575    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:17:27.601869 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:27.601837    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr"]
Oct 30 17:17:27.606209 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:27.606183    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr"
Oct 30 17:17:27.624324 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:27.624298    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-42d381-serving-cert\""
Oct 30 17:17:27.630773 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:27.630735    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d47a5a77-8d4f-401b-b4a3-80c2d27718a9-metrics-certs\") pod \"maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr\" (UID: \"d47a5a77-8d4f-401b-b4a3-80c2d27718a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr"
Oct 30 17:17:27.630985 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:27.630959    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d47a5a77-8d4f-401b-b4a3-80c2d27718a9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr\" (UID: \"d47a5a77-8d4f-401b-b4a3-80c2d27718a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr"
Oct 30 17:17:27.631074 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:27.631000    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt744\" (UniqueName: \"kubernetes.io/projected/d47a5a77-8d4f-401b-b4a3-80c2d27718a9-kube-api-access-zt744\") pod \"maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr\" (UID: \"d47a5a77-8d4f-401b-b4a3-80c2d27718a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr"
Oct 30 17:17:27.631074 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:27.631031    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d47a5a77-8d4f-401b-b4a3-80c2d27718a9-maas-api-tls\") pod \"maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr\" (UID: \"d47a5a77-8d4f-401b-b4a3-80c2d27718a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr"
Oct 30 17:17:27.639485 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:27.639458    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr"]
Oct 30 17:17:27.733022 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:27.732981    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d47a5a77-8d4f-401b-b4a3-80c2d27718a9-metrics-certs\") pod \"maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr\" (UID: \"d47a5a77-8d4f-401b-b4a3-80c2d27718a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr"
Oct 30 17:17:27.733188 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:27.733161    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d47a5a77-8d4f-401b-b4a3-80c2d27718a9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr\" (UID: \"d47a5a77-8d4f-401b-b4a3-80c2d27718a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr"
Oct 30 17:17:27.733249 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:27.733191    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt744\" (UniqueName: \"kubernetes.io/projected/d47a5a77-8d4f-401b-b4a3-80c2d27718a9-kube-api-access-zt744\") pod \"maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr\" (UID: \"d47a5a77-8d4f-401b-b4a3-80c2d27718a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr"
Oct 30 17:17:27.733307 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:27.733260    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d47a5a77-8d4f-401b-b4a3-80c2d27718a9-maas-api-tls\") pod \"maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr\" (UID: \"d47a5a77-8d4f-401b-b4a3-80c2d27718a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr"
Oct 30 17:17:27.733440 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:17:27.733417    4433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-42d381-serving-cert: secret "maas-api-e2e-inf-a-42d381-serving-cert" not found
Oct 30 17:17:27.733534 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:17:27.733506    4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d47a5a77-8d4f-401b-b4a3-80c2d27718a9-maas-api-tls podName:d47a5a77-8d4f-401b-b4a3-80c2d27718a9 nodeName:}" failed. No retries permitted until 2026-10-30 17:17:28.233483198 +0000 UTC m=+1824.668385383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d47a5a77-8d4f-401b-b4a3-80c2d27718a9-maas-api-tls") pod "maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr" (UID: "d47a5a77-8d4f-401b-b4a3-80c2d27718a9") : secret "maas-api-e2e-inf-a-42d381-serving-cert" not found
Oct 30 17:17:27.734141 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:27.734119    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d47a5a77-8d4f-401b-b4a3-80c2d27718a9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr\" (UID: \"d47a5a77-8d4f-401b-b4a3-80c2d27718a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr"
Oct 30 17:17:27.735529 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:27.735508    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d47a5a77-8d4f-401b-b4a3-80c2d27718a9-metrics-certs\") pod \"maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr\" (UID: \"d47a5a77-8d4f-401b-b4a3-80c2d27718a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr"
Oct 30 17:17:27.757215 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:27.757187    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt744\" (UniqueName: \"kubernetes.io/projected/d47a5a77-8d4f-401b-b4a3-80c2d27718a9-kube-api-access-zt744\") pod \"maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr\" (UID: \"d47a5a77-8d4f-401b-b4a3-80c2d27718a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr"
Oct 30 17:17:28.240545 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:28.240501    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d47a5a77-8d4f-401b-b4a3-80c2d27718a9-maas-api-tls\") pod \"maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr\" (UID: \"d47a5a77-8d4f-401b-b4a3-80c2d27718a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr"
Oct 30 17:17:28.242940 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:28.242912    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d47a5a77-8d4f-401b-b4a3-80c2d27718a9-maas-api-tls\") pod \"maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr\" (UID: \"d47a5a77-8d4f-401b-b4a3-80c2d27718a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr"
Oct 30 17:17:28.521939 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:28.521771    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr"
Oct 30 17:17:28.884062 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:28.884036    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr"]
Oct 30 17:17:28.887173 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:17:28.887142    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd47a5a77_8d4f_401b_b4a3_80c2d27718a9.slice/crio-587de0d7512fb83e44663aa22f20046450d1b34a66fdfb77e3bd242d301fe1d7 WatchSource:0}: Error finding container 587de0d7512fb83e44663aa22f20046450d1b34a66fdfb77e3bd242d301fe1d7: Status 404 returned error can't find the container with id 587de0d7512fb83e44663aa22f20046450d1b34a66fdfb77e3bd242d301fe1d7
Oct 30 17:17:29.707105 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:29.707063    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr" event={"ID":"d47a5a77-8d4f-401b-b4a3-80c2d27718a9","Type":"ContainerStarted","Data":"54774d1a67235481922c6a7195c597c91ec65844830283a4bf49c8eb5a6ca5b9"}
Oct 30 17:17:29.707763 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:29.707134    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr" event={"ID":"d47a5a77-8d4f-401b-b4a3-80c2d27718a9","Type":"ContainerStarted","Data":"587de0d7512fb83e44663aa22f20046450d1b34a66fdfb77e3bd242d301fe1d7"}
Oct 30 17:17:29.707763 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:29.707181    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr"
Oct 30 17:17:29.752027 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:29.751910    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr" podStartSLOduration=2.261164137 podStartE2EDuration="2.751890971s" totalImagesPullingTime="490.726834ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:17:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:17:29.748910852 +0000 UTC m=+1826.183813059" watchObservedRunningTime="2026-10-30 17:17:29.751890971 +0000 UTC m=+1826.186793180"
Oct 30 17:17:31.156471 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.156261    4433 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 17:17:31.156471 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.156322    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7" podUID="21dc53b6-74d2-48a6-b5dd-3b1d2940e845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 17:17:31.375793 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.375766    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"
Oct 30 17:17:31.410144 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.410091    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-istiod-ca-cert\") pod \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") "
Oct 30 17:17:31.411147 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.410999    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-istiod-ca-cert" pod "21dc53b6-74d2-48a6-b5dd-3b1d2940e845" (UID: "21dc53b6-74d2-48a6-b5dd-3b1d2940e845"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:17:31.411447 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.411399    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-istio-token\" (UniqueName: \"kubernetes.io/projected/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-istio-token\") pod \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") "
Oct 30 17:17:31.411830 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.411812    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-istio-data\") pod \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") "
Oct 30 17:17:31.412176 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.412152    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-kube-api-access-v84ml\" (UniqueName: \"kubernetes.io/projected/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-kube-api-access-v84ml\") pod \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") "
Oct 30 17:17:31.414761 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.414581    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-istio-data" pod "21dc53b6-74d2-48a6-b5dd-3b1d2940e845" (UID: "21dc53b6-74d2-48a6-b5dd-3b1d2940e845"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:17:31.414986 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.414732    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-workload-socket\") pod \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") "
Oct 30 17:17:31.415150 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.415126    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-workload-socket" pod "21dc53b6-74d2-48a6-b5dd-3b1d2940e845" (UID: "21dc53b6-74d2-48a6-b5dd-3b1d2940e845"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:17:31.415257 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.415241    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-istio-envoy\") pod \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") "
Oct 30 17:17:31.415438 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.415420    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-credential-socket\") pod \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") "
Oct 30 17:17:31.415575 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.415560    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-istio-podinfo\") pod \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") "
Oct 30 17:17:31.415705 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.415688    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-workload-certs\") pod \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\" (UID: \"21dc53b6-74d2-48a6-b5dd-3b1d2940e845\") "
Oct 30 17:17:31.416600 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.416577    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:17:31.416729 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.416705    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:17:31.416729 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.416730    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:17:31.416957 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.416935    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-workload-certs" pod "21dc53b6-74d2-48a6-b5dd-3b1d2940e845" (UID: "21dc53b6-74d2-48a6-b5dd-3b1d2940e845"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:17:31.417398 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.417140    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-credential-socket" pod "21dc53b6-74d2-48a6-b5dd-3b1d2940e845" (UID: "21dc53b6-74d2-48a6-b5dd-3b1d2940e845"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:17:31.420710 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.420674    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-kube-api-access-v84ml" pod "21dc53b6-74d2-48a6-b5dd-3b1d2940e845" (UID: "21dc53b6-74d2-48a6-b5dd-3b1d2940e845"). InnerVolumeSpecName "kube-api-access-v84ml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:17:31.420923 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.420896    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-istio-token" pod "21dc53b6-74d2-48a6-b5dd-3b1d2940e845" (UID: "21dc53b6-74d2-48a6-b5dd-3b1d2940e845"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:17:31.423591 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.423563    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-istio-podinfo" pod "21dc53b6-74d2-48a6-b5dd-3b1d2940e845" (UID: "21dc53b6-74d2-48a6-b5dd-3b1d2940e845"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:17:31.423591 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.423582    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-istio-envoy" pod "21dc53b6-74d2-48a6-b5dd-3b1d2940e845" (UID: "21dc53b6-74d2-48a6-b5dd-3b1d2940e845"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:17:31.518077 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.517994    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:17:31.518077 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.518025    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v84ml\" (UniqueName: \"kubernetes.io/projected/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-kube-api-access-v84ml\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:17:31.518077 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.518037    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:17:31.518077 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.518046    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:17:31.518077 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.518055    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:17:31.518077 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.518065    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21dc53b6-74d2-48a6-b5dd-3b1d2940e845-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:17:31.720924 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.720888    4433 generic.go:356] "Generic (PLEG): container finished" podID="21dc53b6-74d2-48a6-b5dd-3b1d2940e845" containerID="244f946d18f2dc54e9e26f3fd19380cfc225c9862e2096f4fda2778eda487017" exitCode=0
Oct 30 17:17:31.721087 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.720938    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"
Oct 30 17:17:31.721087 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.720957    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7" event={"ID":"21dc53b6-74d2-48a6-b5dd-3b1d2940e845","Type":"ContainerDied","Data":"244f946d18f2dc54e9e26f3fd19380cfc225c9862e2096f4fda2778eda487017"}
Oct 30 17:17:31.721087 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.721011    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7" event={"ID":"21dc53b6-74d2-48a6-b5dd-3b1d2940e845","Type":"ContainerDied","Data":"396b4989f2800e556596fc4c9ef8e59e3a62b1fa79884e0378e1e43c53ec93c4"}
Oct 30 17:17:31.721087 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.721037    4433 scope.go:122] "RemoveContainer" containerID="244f946d18f2dc54e9e26f3fd19380cfc225c9862e2096f4fda2778eda487017"
Oct 30 17:17:31.731676 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.731652    4433 scope.go:122] "RemoveContainer" containerID="244f946d18f2dc54e9e26f3fd19380cfc225c9862e2096f4fda2778eda487017"
Oct 30 17:17:31.732013 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:17:31.731984    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"244f946d18f2dc54e9e26f3fd19380cfc225c9862e2096f4fda2778eda487017\": container with ID starting with 244f946d18f2dc54e9e26f3fd19380cfc225c9862e2096f4fda2778eda487017 not found: ID does not exist" containerID="244f946d18f2dc54e9e26f3fd19380cfc225c9862e2096f4fda2778eda487017"
Oct 30 17:17:31.732097 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.732024    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"244f946d18f2dc54e9e26f3fd19380cfc225c9862e2096f4fda2778eda487017"} err="failed to get container status \"244f946d18f2dc54e9e26f3fd19380cfc225c9862e2096f4fda2778eda487017\": rpc error: code = NotFound desc = could not find container \"244f946d18f2dc54e9e26f3fd19380cfc225c9862e2096f4fda2778eda487017\": container with ID starting with 244f946d18f2dc54e9e26f3fd19380cfc225c9862e2096f4fda2778eda487017 not found: ID does not exist"
Oct 30 17:17:31.775612 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.775526    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"]
Oct 30 17:17:31.787861 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:31.787825    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-hzmb7"]
Oct 30 17:17:33.697531 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:33.697489    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21dc53b6-74d2-48a6-b5dd-3b1d2940e845" path="/var/lib/kubelet/pods/21dc53b6-74d2-48a6-b5dd-3b1d2940e845/volumes"
Oct 30 17:17:35.719707 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:35.719676    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr"
Oct 30 17:17:37.508389 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:37.508354    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64"]
Oct 30 17:17:37.509347 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:37.509318    4433 cpu_manager.go:414] "containerMap: removing container" podUID="21dc53b6-74d2-48a6-b5dd-3b1d2940e845" containerName="istio-proxy"
Oct 30 17:17:37.509347 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:37.509347    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21dc53b6-74d2-48a6-b5dd-3b1d2940e845" containerName="istio-proxy"
Oct 30 17:17:37.509591 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:37.509557    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="21dc53b6-74d2-48a6-b5dd-3b1d2940e845" containerName="istio-proxy"
Oct 30 17:17:37.516491 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:37.516466    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64"
Oct 30 17:17:37.536878 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:37.536845    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64"]
Oct 30 17:17:37.543159 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:37.543131    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-78c9c2\"/\"openshift-service-ca.crt\""
Oct 30 17:17:37.543314 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:37.543241    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-78c9c2\"/\"auth-test-model-78c9c2-kserve-self-signed-certs\""
Oct 30 17:17:37.543314 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:37.543306    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-78c9c2\"/\"kube-root-ca.crt\""
Oct 30 17:17:37.543605 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:37.543585    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-78c9c2\"/\"default-dockercfg-4ll8h\""
Oct 30 17:17:37.616098 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:37.616062    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f6eaf54-07a2-4010-9b79-e82670f15262-dshm\") pod \"auth-test-model-78c9c2-kserve-79756ff877-vbs64\" (UID: \"3f6eaf54-07a2-4010-9b79-e82670f15262\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64"
Oct 30 17:17:37.616098 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:37.616104    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pbc4\" (UniqueName: \"kubernetes.io/projected/3f6eaf54-07a2-4010-9b79-e82670f15262-kube-api-access-7pbc4\") pod \"auth-test-model-78c9c2-kserve-79756ff877-vbs64\" (UID: \"3f6eaf54-07a2-4010-9b79-e82670f15262\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64"
Oct 30 17:17:37.616359 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:37.616232    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f6eaf54-07a2-4010-9b79-e82670f15262-home\") pod \"auth-test-model-78c9c2-kserve-79756ff877-vbs64\" (UID: \"3f6eaf54-07a2-4010-9b79-e82670f15262\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64"
Oct 30 17:17:37.616359 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:37.616262    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f6eaf54-07a2-4010-9b79-e82670f15262-tls-certs\") pod \"auth-test-model-78c9c2-kserve-79756ff877-vbs64\" (UID: \"3f6eaf54-07a2-4010-9b79-e82670f15262\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64"
Oct 30 17:17:37.616522 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:37.616363    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f6eaf54-07a2-4010-9b79-e82670f15262-model-cache\") pod \"auth-test-model-78c9c2-kserve-79756ff877-vbs64\" (UID: \"3f6eaf54-07a2-4010-9b79-e82670f15262\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64"
Oct 30 17:17:37.616522 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:37.616390    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f6eaf54-07a2-4010-9b79-e82670f15262-tmp-dir\") pod \"auth-test-model-78c9c2-kserve-79756ff877-vbs64\" (UID: \"3f6eaf54-07a2-4010-9b79-e82670f15262\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64"
Oct 30 17:17:37.718338 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:37.718279    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f6eaf54-07a2-4010-9b79-e82670f15262-tmp-dir\") pod \"auth-test-model-78c9c2-kserve-79756ff877-vbs64\" (UID: \"3f6eaf54-07a2-4010-9b79-e82670f15262\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64"
Oct 30 17:17:37.718610 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:37.718591    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f6eaf54-07a2-4010-9b79-e82670f15262-dshm\") pod \"auth-test-model-78c9c2-kserve-79756ff877-vbs64\" (UID: \"3f6eaf54-07a2-4010-9b79-e82670f15262\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64"
Oct 30 17:17:37.718677 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:37.718635    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pbc4\" (UniqueName: \"kubernetes.io/projected/3f6eaf54-07a2-4010-9b79-e82670f15262-kube-api-access-7pbc4\") pod \"auth-test-model-78c9c2-kserve-79756ff877-vbs64\" (UID: \"3f6eaf54-07a2-4010-9b79-e82670f15262\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64"
Oct 30 17:17:37.718742 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:37.718718    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f6eaf54-07a2-4010-9b79-e82670f15262-tmp-dir\") pod \"auth-test-model-78c9c2-kserve-79756ff877-vbs64\" (UID: \"3f6eaf54-07a2-4010-9b79-e82670f15262\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64"
Oct 30 17:17:37.718799 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:37.718738    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f6eaf54-07a2-4010-9b79-e82670f15262-home\") pod \"auth-test-model-78c9c2-kserve-79756ff877-vbs64\" (UID: \"3f6eaf54-07a2-4010-9b79-e82670f15262\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64"
Oct 30 17:17:37.718799 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:37.718791    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f6eaf54-07a2-4010-9b79-e82670f15262-tls-certs\") pod \"auth-test-model-78c9c2-kserve-79756ff877-vbs64\" (UID: \"3f6eaf54-07a2-4010-9b79-e82670f15262\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64"
Oct 30 17:17:37.718898 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:37.718876    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f6eaf54-07a2-4010-9b79-e82670f15262-model-cache\") pod \"auth-test-model-78c9c2-kserve-79756ff877-vbs64\" (UID: \"3f6eaf54-07a2-4010-9b79-e82670f15262\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64"
Oct 30 17:17:37.719065 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:37.719044    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f6eaf54-07a2-4010-9b79-e82670f15262-home\") pod \"auth-test-model-78c9c2-kserve-79756ff877-vbs64\" (UID: \"3f6eaf54-07a2-4010-9b79-e82670f15262\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64"
Oct 30 17:17:37.719238 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:37.719214    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f6eaf54-07a2-4010-9b79-e82670f15262-model-cache\") pod \"auth-test-model-78c9c2-kserve-79756ff877-vbs64\" (UID: \"3f6eaf54-07a2-4010-9b79-e82670f15262\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64"
Oct 30 17:17:37.721249 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:37.721220    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f6eaf54-07a2-4010-9b79-e82670f15262-dshm\") pod \"auth-test-model-78c9c2-kserve-79756ff877-vbs64\" (UID: \"3f6eaf54-07a2-4010-9b79-e82670f15262\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64"
Oct 30 17:17:37.721443 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:37.721386    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f6eaf54-07a2-4010-9b79-e82670f15262-tls-certs\") pod \"auth-test-model-78c9c2-kserve-79756ff877-vbs64\" (UID: \"3f6eaf54-07a2-4010-9b79-e82670f15262\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64"
Oct 30 17:17:37.743670 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:37.743645    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pbc4\" (UniqueName: \"kubernetes.io/projected/3f6eaf54-07a2-4010-9b79-e82670f15262-kube-api-access-7pbc4\") pod \"auth-test-model-78c9c2-kserve-79756ff877-vbs64\" (UID: \"3f6eaf54-07a2-4010-9b79-e82670f15262\") " pod="ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64"
Oct 30 17:17:37.842481 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:37.841883    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64"
Oct 30 17:17:38.299741 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:38.299711    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64"]
Oct 30 17:17:38.302453 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:17:38.302392    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f6eaf54_07a2_4010_9b79_e82670f15262.slice/crio-d0aa33442297c1340c28dab02378c4e3456da05430d1c324e379b1f6daf4ade0 WatchSource:0}: Error finding container d0aa33442297c1340c28dab02378c4e3456da05430d1c324e379b1f6daf4ade0: Status 404 returned error can't find the container with id d0aa33442297c1340c28dab02378c4e3456da05430d1c324e379b1f6daf4ade0
Oct 30 17:17:38.772013 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:38.771973    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64" event={"ID":"3f6eaf54-07a2-4010-9b79-e82670f15262","Type":"ContainerStarted","Data":"09fb7757e785afe73e877c523ea1d706aec5c7764fba9c6fd71e5d622123d9ff"}
Oct 30 17:17:38.772587 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:38.772024    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64" event={"ID":"3f6eaf54-07a2-4010-9b79-e82670f15262","Type":"ContainerStarted","Data":"d0aa33442297c1340c28dab02378c4e3456da05430d1c324e379b1f6daf4ade0"}
Oct 30 17:17:38.816634 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:38.816576    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64" podStartSLOduration=1.816560572 podStartE2EDuration="1.816560572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:17:38.814742473 +0000 UTC m=+1835.249644675" watchObservedRunningTime="2026-10-30 17:17:38.816560572 +0000 UTC m=+1835.251462778"
Oct 30 17:17:38.842848 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:38.842818    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64"
Oct 30 17:17:40.566001 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:40.565960    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"]
Oct 30 17:17:40.566477 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:40.566221    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8" podUID="03c0ed14-97bb-4f58-9529-19ee6549b8c9" containerName="istio-proxy" containerID="cri-o://a9e257a60f17cda250fa3ca42e55323ee7656692b8f1f33c9c1c7074132e7d7d" gracePeriod=30
Oct 30 17:17:44.507302 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:44.507263    4433 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:17:44.507737 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:44.507327    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8" podUID="03c0ed14-97bb-4f58-9529-19ee6549b8c9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:17:45.843497 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:45.841503    4433 generic.go:356] "Generic (PLEG): container finished" podID="03c0ed14-97bb-4f58-9529-19ee6549b8c9" containerID="a9e257a60f17cda250fa3ca42e55323ee7656692b8f1f33c9c1c7074132e7d7d" exitCode=0
Oct 30 17:17:45.843497 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:45.841594    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8" event={"ID":"03c0ed14-97bb-4f58-9529-19ee6549b8c9","Type":"ContainerDied","Data":"a9e257a60f17cda250fa3ca42e55323ee7656692b8f1f33c9c1c7074132e7d7d"}
Oct 30 17:17:45.919503 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:45.919470    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"
Oct 30 17:17:46.064460 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:46.064423    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03c0ed14-97bb-4f58-9529-19ee6549b8c9-kube-api-access-ngfqw\" (UniqueName: \"kubernetes.io/projected/03c0ed14-97bb-4f58-9529-19ee6549b8c9-kube-api-access-ngfqw\") pod \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") "
Oct 30 17:17:46.064663 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:46.064483    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03c0ed14-97bb-4f58-9529-19ee6549b8c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c0ed14-97bb-4f58-9529-19ee6549b8c9-workload-socket\") pod \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") "
Oct 30 17:17:46.064663 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:46.064526    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03c0ed14-97bb-4f58-9529-19ee6549b8c9-istio-token\" (UniqueName: \"kubernetes.io/projected/03c0ed14-97bb-4f58-9529-19ee6549b8c9-istio-token\") pod \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") "
Oct 30 17:17:46.064663 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:46.064555    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03c0ed14-97bb-4f58-9529-19ee6549b8c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c0ed14-97bb-4f58-9529-19ee6549b8c9-credential-socket\") pod \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") "
Oct 30 17:17:46.064663 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:46.064611    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03c0ed14-97bb-4f58-9529-19ee6549b8c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03c0ed14-97bb-4f58-9529-19ee6549b8c9-istio-envoy\") pod \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") "
Oct 30 17:17:46.064663 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:46.064661    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03c0ed14-97bb-4f58-9529-19ee6549b8c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03c0ed14-97bb-4f58-9529-19ee6549b8c9-istio-podinfo\") pod \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") "
Oct 30 17:17:46.064942 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:46.064780    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03c0ed14-97bb-4f58-9529-19ee6549b8c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03c0ed14-97bb-4f58-9529-19ee6549b8c9-workload-certs\") pod \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") "
Oct 30 17:17:46.064942 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:46.064806    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03c0ed14-97bb-4f58-9529-19ee6549b8c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03c0ed14-97bb-4f58-9529-19ee6549b8c9-istio-data\") pod \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") "
Oct 30 17:17:46.064942 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:46.064898    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03c0ed14-97bb-4f58-9529-19ee6549b8c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03c0ed14-97bb-4f58-9529-19ee6549b8c9-istiod-ca-cert\") pod \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\" (UID: \"03c0ed14-97bb-4f58-9529-19ee6549b8c9\") "
Oct 30 17:17:46.066614 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:46.066313    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c0ed14-97bb-4f58-9529-19ee6549b8c9-credential-socket" pod "03c0ed14-97bb-4f58-9529-19ee6549b8c9" (UID: "03c0ed14-97bb-4f58-9529-19ee6549b8c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:17:46.066614 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:46.066356    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c0ed14-97bb-4f58-9529-19ee6549b8c9-workload-certs" pod "03c0ed14-97bb-4f58-9529-19ee6549b8c9" (UID: "03c0ed14-97bb-4f58-9529-19ee6549b8c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:17:46.066837 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:46.066772    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03c0ed14-97bb-4f58-9529-19ee6549b8c9-istiod-ca-cert" pod "03c0ed14-97bb-4f58-9529-19ee6549b8c9" (UID: "03c0ed14-97bb-4f58-9529-19ee6549b8c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:17:46.068129 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:46.068057    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c0ed14-97bb-4f58-9529-19ee6549b8c9-workload-socket" pod "03c0ed14-97bb-4f58-9529-19ee6549b8c9" (UID: "03c0ed14-97bb-4f58-9529-19ee6549b8c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:17:46.068422 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:46.068371    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c0ed14-97bb-4f58-9529-19ee6549b8c9-istio-data" pod "03c0ed14-97bb-4f58-9529-19ee6549b8c9" (UID: "03c0ed14-97bb-4f58-9529-19ee6549b8c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:17:46.075430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:46.070548    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03c0ed14-97bb-4f58-9529-19ee6549b8c9-kube-api-access-ngfqw" pod "03c0ed14-97bb-4f58-9529-19ee6549b8c9" (UID: "03c0ed14-97bb-4f58-9529-19ee6549b8c9"). InnerVolumeSpecName "kube-api-access-ngfqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:17:46.075430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:46.070625    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c0ed14-97bb-4f58-9529-19ee6549b8c9-istio-envoy" pod "03c0ed14-97bb-4f58-9529-19ee6549b8c9" (UID: "03c0ed14-97bb-4f58-9529-19ee6549b8c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:17:46.075430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:46.070709    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03c0ed14-97bb-4f58-9529-19ee6549b8c9-istio-token" pod "03c0ed14-97bb-4f58-9529-19ee6549b8c9" (UID: "03c0ed14-97bb-4f58-9529-19ee6549b8c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:17:46.075430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:46.070917    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03c0ed14-97bb-4f58-9529-19ee6549b8c9-istio-podinfo" pod "03c0ed14-97bb-4f58-9529-19ee6549b8c9" (UID: "03c0ed14-97bb-4f58-9529-19ee6549b8c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:17:46.167886 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:46.167798    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03c0ed14-97bb-4f58-9529-19ee6549b8c9-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:17:46.167886 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:46.167830    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03c0ed14-97bb-4f58-9529-19ee6549b8c9-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:17:46.167886 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:46.167839    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03c0ed14-97bb-4f58-9529-19ee6549b8c9-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:17:46.167886 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:46.167854    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngfqw\" (UniqueName: \"kubernetes.io/projected/03c0ed14-97bb-4f58-9529-19ee6549b8c9-kube-api-access-ngfqw\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:17:46.167886 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:46.167863    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c0ed14-97bb-4f58-9529-19ee6549b8c9-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:17:46.167886 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:46.167871    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03c0ed14-97bb-4f58-9529-19ee6549b8c9-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:17:46.167886 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:46.167879    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c0ed14-97bb-4f58-9529-19ee6549b8c9-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:17:46.167886 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:46.167888    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03c0ed14-97bb-4f58-9529-19ee6549b8c9-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:17:46.168248 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:46.167896    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03c0ed14-97bb-4f58-9529-19ee6549b8c9-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:17:46.849290 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:46.849260    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"
Oct 30 17:17:46.849290 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:46.849270    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8" event={"ID":"03c0ed14-97bb-4f58-9529-19ee6549b8c9","Type":"ContainerDied","Data":"aeeb013791033ba88ac7431be13fbb9e5e5da87b7d788b15149e6a894e078e10"}
Oct 30 17:17:46.849769 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:46.849338    4433 scope.go:122] "RemoveContainer" containerID="a9e257a60f17cda250fa3ca42e55323ee7656692b8f1f33c9c1c7074132e7d7d"
Oct 30 17:17:46.913734 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:46.913705    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"]
Oct 30 17:17:46.925418 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:46.925376    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-xlxw8"]
Oct 30 17:17:47.697322 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:47.697285    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03c0ed14-97bb-4f58-9529-19ee6549b8c9" path="/var/lib/kubelet/pods/03c0ed14-97bb-4f58-9529-19ee6549b8c9/volumes"
Oct 30 17:17:47.842775 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:47.842739    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64"
Oct 30 17:17:47.855218 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:47.855192    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64"
Oct 30 17:17:47.869818 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:47.869792    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64"
Oct 30 17:17:48.821426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:48.821302    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:17:49.190578 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:49.190540    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd"]
Oct 30 17:17:49.191388 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:49.191364    4433 cpu_manager.go:414] "containerMap: removing container" podUID="03c0ed14-97bb-4f58-9529-19ee6549b8c9" containerName="istio-proxy"
Oct 30 17:17:49.191517 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:49.191389    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03c0ed14-97bb-4f58-9529-19ee6549b8c9" containerName="istio-proxy"
Oct 30 17:17:49.191590 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:49.191573    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="03c0ed14-97bb-4f58-9529-19ee6549b8c9" containerName="istio-proxy"
Oct 30 17:17:49.196863 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:49.196844    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd"
Oct 30 17:17:49.216805 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:49.216774    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-987dfc-serving-cert\""
Oct 30 17:17:49.224709 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:49.224673    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd"]
Oct 30 17:17:49.326815 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:49.326780    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cd24c86-aae9-453b-a7a7-23296a87c420-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd\" (UID: \"1cd24c86-aae9-453b-a7a7-23296a87c420\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd"
Oct 30 17:17:49.326989 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:49.326841    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd24c86-aae9-453b-a7a7-23296a87c420-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd\" (UID: \"1cd24c86-aae9-453b-a7a7-23296a87c420\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd"
Oct 30 17:17:49.326989 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:49.326923    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cd24c86-aae9-453b-a7a7-23296a87c420-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd\" (UID: \"1cd24c86-aae9-453b-a7a7-23296a87c420\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd"
Oct 30 17:17:49.327180 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:49.327163    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpqs9\" (UniqueName: \"kubernetes.io/projected/1cd24c86-aae9-453b-a7a7-23296a87c420-kube-api-access-hpqs9\") pod \"maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd\" (UID: \"1cd24c86-aae9-453b-a7a7-23296a87c420\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd"
Oct 30 17:17:49.431627 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:49.431572    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpqs9\" (UniqueName: \"kubernetes.io/projected/1cd24c86-aae9-453b-a7a7-23296a87c420-kube-api-access-hpqs9\") pod \"maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd\" (UID: \"1cd24c86-aae9-453b-a7a7-23296a87c420\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd"
Oct 30 17:17:49.431991 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:49.431951    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cd24c86-aae9-453b-a7a7-23296a87c420-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd\" (UID: \"1cd24c86-aae9-453b-a7a7-23296a87c420\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd"
Oct 30 17:17:49.432150 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:49.432028    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd24c86-aae9-453b-a7a7-23296a87c420-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd\" (UID: \"1cd24c86-aae9-453b-a7a7-23296a87c420\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd"
Oct 30 17:17:49.432150 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:49.432063    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cd24c86-aae9-453b-a7a7-23296a87c420-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd\" (UID: \"1cd24c86-aae9-453b-a7a7-23296a87c420\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd"
Oct 30 17:17:49.433222 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:49.433194    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd24c86-aae9-453b-a7a7-23296a87c420-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd\" (UID: \"1cd24c86-aae9-453b-a7a7-23296a87c420\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd"
Oct 30 17:17:49.440389 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:49.440142    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cd24c86-aae9-453b-a7a7-23296a87c420-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd\" (UID: \"1cd24c86-aae9-453b-a7a7-23296a87c420\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd"
Oct 30 17:17:49.440734 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:49.440671    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cd24c86-aae9-453b-a7a7-23296a87c420-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd\" (UID: \"1cd24c86-aae9-453b-a7a7-23296a87c420\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd"
Oct 30 17:17:49.455370 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:49.455332    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpqs9\" (UniqueName: \"kubernetes.io/projected/1cd24c86-aae9-453b-a7a7-23296a87c420-kube-api-access-hpqs9\") pod \"maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd\" (UID: \"1cd24c86-aae9-453b-a7a7-23296a87c420\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd"
Oct 30 17:17:49.511552 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:49.511513    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd"
Oct 30 17:17:49.835063 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:49.835033    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd"]
Oct 30 17:17:49.837759 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:17:49.837726    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd24c86_aae9_453b_a7a7_23296a87c420.slice/crio-21f8d96fa819b5d1332880dbb0b3c56cb510cc18a64f3d1b48a3470d55bb5024 WatchSource:0}: Error finding container 21f8d96fa819b5d1332880dbb0b3c56cb510cc18a64f3d1b48a3470d55bb5024: Status 404 returned error can't find the container with id 21f8d96fa819b5d1332880dbb0b3c56cb510cc18a64f3d1b48a3470d55bb5024
Oct 30 17:17:49.882907 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:49.882873    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd" event={"ID":"1cd24c86-aae9-453b-a7a7-23296a87c420","Type":"ContainerStarted","Data":"21f8d96fa819b5d1332880dbb0b3c56cb510cc18a64f3d1b48a3470d55bb5024"}
Oct 30 17:17:50.895926 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:50.895884    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd" event={"ID":"1cd24c86-aae9-453b-a7a7-23296a87c420","Type":"ContainerStarted","Data":"ee907c4024aead38c212dcf43378fa065e6822c9a066adf40e9efb729aeac24a"}
Oct 30 17:17:50.896800 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:50.896775    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd"
Oct 30 17:17:50.952562 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:50.950301    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd" podStartSLOduration=1.447283992 podStartE2EDuration="1.950275589s" totalImagesPullingTime="502.991597ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:17:50.941029686 +0000 UTC m=+1847.375931896" watchObservedRunningTime="2026-10-30 17:17:50.950275589 +0000 UTC m=+1847.385177795"
Oct 30 17:17:56.909465 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:17:56.909420    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd"
Oct 30 17:17:56.984857 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:17:56.984823    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:18:21.766380 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:21.766334    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq"]
Oct 30 17:18:21.772428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:21.772390    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq"
Oct 30 17:18:21.802492 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:21.802460    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7c52f8\"/\"kube-root-ca.crt\""
Oct 30 17:18:21.802492 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:21.802466    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7c52f8\"/\"test-model-7c52f8-kserve-self-signed-certs\""
Oct 30 17:18:21.802886 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:21.802856    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7c52f8\"/\"openshift-service-ca.crt\""
Oct 30 17:18:21.803314 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:21.803296    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7c52f8\"/\"default-dockercfg-2pllk\""
Oct 30 17:18:21.808472 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:21.808443    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq"]
Oct 30 17:18:21.898530 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:21.898491    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd03e357-fdbb-438b-847e-26ba74b88949-home\") pod \"test-model-7c52f8-kserve-5c9f8d47df-rsxwq\" (UID: \"dd03e357-fdbb-438b-847e-26ba74b88949\") " pod="ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq"
Oct 30 17:18:21.898701 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:21.898539    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdr2t\" (UniqueName: \"kubernetes.io/projected/dd03e357-fdbb-438b-847e-26ba74b88949-kube-api-access-kdr2t\") pod \"test-model-7c52f8-kserve-5c9f8d47df-rsxwq\" (UID: \"dd03e357-fdbb-438b-847e-26ba74b88949\") " pod="ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq"
Oct 30 17:18:21.898748 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:21.898713    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd03e357-fdbb-438b-847e-26ba74b88949-model-cache\") pod \"test-model-7c52f8-kserve-5c9f8d47df-rsxwq\" (UID: \"dd03e357-fdbb-438b-847e-26ba74b88949\") " pod="ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq"
Oct 30 17:18:21.898799 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:21.898785    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd03e357-fdbb-438b-847e-26ba74b88949-tmp-dir\") pod \"test-model-7c52f8-kserve-5c9f8d47df-rsxwq\" (UID: \"dd03e357-fdbb-438b-847e-26ba74b88949\") " pod="ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq"
Oct 30 17:18:21.898953 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:21.898933    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd03e357-fdbb-438b-847e-26ba74b88949-dshm\") pod \"test-model-7c52f8-kserve-5c9f8d47df-rsxwq\" (UID: \"dd03e357-fdbb-438b-847e-26ba74b88949\") " pod="ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq"
Oct 30 17:18:21.899058 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:21.899041    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd03e357-fdbb-438b-847e-26ba74b88949-tls-certs\") pod \"test-model-7c52f8-kserve-5c9f8d47df-rsxwq\" (UID: \"dd03e357-fdbb-438b-847e-26ba74b88949\") " pod="ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq"
Oct 30 17:18:22.005795 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:22.005751    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd03e357-fdbb-438b-847e-26ba74b88949-model-cache\") pod \"test-model-7c52f8-kserve-5c9f8d47df-rsxwq\" (UID: \"dd03e357-fdbb-438b-847e-26ba74b88949\") " pod="ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq"
Oct 30 17:18:22.006102 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:22.006077    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd03e357-fdbb-438b-847e-26ba74b88949-tmp-dir\") pod \"test-model-7c52f8-kserve-5c9f8d47df-rsxwq\" (UID: \"dd03e357-fdbb-438b-847e-26ba74b88949\") " pod="ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq"
Oct 30 17:18:22.006394 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:22.006374    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd03e357-fdbb-438b-847e-26ba74b88949-dshm\") pod \"test-model-7c52f8-kserve-5c9f8d47df-rsxwq\" (UID: \"dd03e357-fdbb-438b-847e-26ba74b88949\") " pod="ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq"
Oct 30 17:18:22.006610 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:22.006591    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd03e357-fdbb-438b-847e-26ba74b88949-tls-certs\") pod \"test-model-7c52f8-kserve-5c9f8d47df-rsxwq\" (UID: \"dd03e357-fdbb-438b-847e-26ba74b88949\") " pod="ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq"
Oct 30 17:18:22.006770 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:22.006751    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd03e357-fdbb-438b-847e-26ba74b88949-home\") pod \"test-model-7c52f8-kserve-5c9f8d47df-rsxwq\" (UID: \"dd03e357-fdbb-438b-847e-26ba74b88949\") " pod="ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq"
Oct 30 17:18:22.006878 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:22.006863    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdr2t\" (UniqueName: \"kubernetes.io/projected/dd03e357-fdbb-438b-847e-26ba74b88949-kube-api-access-kdr2t\") pod \"test-model-7c52f8-kserve-5c9f8d47df-rsxwq\" (UID: \"dd03e357-fdbb-438b-847e-26ba74b88949\") " pod="ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq"
Oct 30 17:18:22.010120 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:22.010090    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd03e357-fdbb-438b-847e-26ba74b88949-home\") pod \"test-model-7c52f8-kserve-5c9f8d47df-rsxwq\" (UID: \"dd03e357-fdbb-438b-847e-26ba74b88949\") " pod="ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq"
Oct 30 17:18:22.010588 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:22.010567    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd03e357-fdbb-438b-847e-26ba74b88949-tmp-dir\") pod \"test-model-7c52f8-kserve-5c9f8d47df-rsxwq\" (UID: \"dd03e357-fdbb-438b-847e-26ba74b88949\") " pod="ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq"
Oct 30 17:18:22.056860 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:22.016643    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd03e357-fdbb-438b-847e-26ba74b88949-model-cache\") pod \"test-model-7c52f8-kserve-5c9f8d47df-rsxwq\" (UID: \"dd03e357-fdbb-438b-847e-26ba74b88949\") " pod="ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq"
Oct 30 17:18:22.056860 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:22.026460    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd03e357-fdbb-438b-847e-26ba74b88949-tls-certs\") pod \"test-model-7c52f8-kserve-5c9f8d47df-rsxwq\" (UID: \"dd03e357-fdbb-438b-847e-26ba74b88949\") " pod="ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq"
Oct 30 17:18:22.056860 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:22.044769    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdr2t\" (UniqueName: \"kubernetes.io/projected/dd03e357-fdbb-438b-847e-26ba74b88949-kube-api-access-kdr2t\") pod \"test-model-7c52f8-kserve-5c9f8d47df-rsxwq\" (UID: \"dd03e357-fdbb-438b-847e-26ba74b88949\") " pod="ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq"
Oct 30 17:18:22.072209 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:22.072176    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd03e357-fdbb-438b-847e-26ba74b88949-dshm\") pod \"test-model-7c52f8-kserve-5c9f8d47df-rsxwq\" (UID: \"dd03e357-fdbb-438b-847e-26ba74b88949\") " pod="ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq"
Oct 30 17:18:22.095740 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:22.095704    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq"
Oct 30 17:18:22.458532 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:22.458501    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq"]
Oct 30 17:18:23.159019 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:23.158981    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq" event={"ID":"dd03e357-fdbb-438b-847e-26ba74b88949","Type":"ContainerStarted","Data":"4c52296e86cd72cef01ceb8425e710e1ff1334e1887ac74a6acd48ba75101df5"}
Oct 30 17:18:23.159019 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:23.159021    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq" event={"ID":"dd03e357-fdbb-438b-847e-26ba74b88949","Type":"ContainerStarted","Data":"2f327d657aacf3920a082880aefa27009b4123088c519b1e54d3642e6e86ac07"}
Oct 30 17:18:23.208015 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:23.207958    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq" podStartSLOduration=2.207940468 podStartE2EDuration="2.207940468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:18:23.205707489 +0000 UTC m=+1879.640609761" watchObservedRunningTime="2026-10-30 17:18:23.207940468 +0000 UTC m=+1879.642842673"
Oct 30 17:18:24.097094 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:24.097053    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq"
Oct 30 17:18:26.979142 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:18:26.979107    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:18:32.096234 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:32.096198    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq"
Oct 30 17:18:32.108939 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:32.108916    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq"
Oct 30 17:18:32.231533 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:32.231496    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq"
Oct 30 17:18:44.483561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:44.483516    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86"]
Oct 30 17:18:44.488848 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:44.488823    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86"
Oct 30 17:18:44.515180 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:44.515145    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-dd5c8f\"/\"kube-root-ca.crt\""
Oct 30 17:18:44.515392 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:44.515277    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-dd5c8f\"/\"default-dockercfg-9bgc5\""
Oct 30 17:18:44.515392 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:44.515336    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-dd5c8f\"/\"openshift-service-ca.crt\""
Oct 30 17:18:44.515641 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:44.515594    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-dd5c8f\"/\"test-model-dd5c8f-kserve-self-signed-certs\""
Oct 30 17:18:44.521434 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:44.521389    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86"]
Oct 30 17:18:44.572415 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:44.572370    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5420c28a-0d07-47df-8d12-1ec97ec9707e-home\") pod \"test-model-dd5c8f-kserve-554d64c6dc-6vm86\" (UID: \"5420c28a-0d07-47df-8d12-1ec97ec9707e\") " pod="ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86"
Oct 30 17:18:44.572588 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:44.572502    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrvgf\" (UniqueName: \"kubernetes.io/projected/5420c28a-0d07-47df-8d12-1ec97ec9707e-kube-api-access-xrvgf\") pod \"test-model-dd5c8f-kserve-554d64c6dc-6vm86\" (UID: \"5420c28a-0d07-47df-8d12-1ec97ec9707e\") " pod="ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86"
Oct 30 17:18:44.572588 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:44.572554    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5420c28a-0d07-47df-8d12-1ec97ec9707e-tls-certs\") pod \"test-model-dd5c8f-kserve-554d64c6dc-6vm86\" (UID: \"5420c28a-0d07-47df-8d12-1ec97ec9707e\") " pod="ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86"
Oct 30 17:18:44.572686 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:44.572637    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5420c28a-0d07-47df-8d12-1ec97ec9707e-model-cache\") pod \"test-model-dd5c8f-kserve-554d64c6dc-6vm86\" (UID: \"5420c28a-0d07-47df-8d12-1ec97ec9707e\") " pod="ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86"
Oct 30 17:18:44.572731 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:44.572722    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5420c28a-0d07-47df-8d12-1ec97ec9707e-tmp-dir\") pod \"test-model-dd5c8f-kserve-554d64c6dc-6vm86\" (UID: \"5420c28a-0d07-47df-8d12-1ec97ec9707e\") " pod="ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86"
Oct 30 17:18:44.572864 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:44.572845    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5420c28a-0d07-47df-8d12-1ec97ec9707e-dshm\") pod \"test-model-dd5c8f-kserve-554d64c6dc-6vm86\" (UID: \"5420c28a-0d07-47df-8d12-1ec97ec9707e\") " pod="ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86"
Oct 30 17:18:44.674552 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:44.674515    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5420c28a-0d07-47df-8d12-1ec97ec9707e-home\") pod \"test-model-dd5c8f-kserve-554d64c6dc-6vm86\" (UID: \"5420c28a-0d07-47df-8d12-1ec97ec9707e\") " pod="ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86"
Oct 30 17:18:44.674759 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:44.674580    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrvgf\" (UniqueName: \"kubernetes.io/projected/5420c28a-0d07-47df-8d12-1ec97ec9707e-kube-api-access-xrvgf\") pod \"test-model-dd5c8f-kserve-554d64c6dc-6vm86\" (UID: \"5420c28a-0d07-47df-8d12-1ec97ec9707e\") " pod="ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86"
Oct 30 17:18:44.674759 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:44.674603    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5420c28a-0d07-47df-8d12-1ec97ec9707e-tls-certs\") pod \"test-model-dd5c8f-kserve-554d64c6dc-6vm86\" (UID: \"5420c28a-0d07-47df-8d12-1ec97ec9707e\") " pod="ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86"
Oct 30 17:18:44.674759 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:44.674641    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5420c28a-0d07-47df-8d12-1ec97ec9707e-model-cache\") pod \"test-model-dd5c8f-kserve-554d64c6dc-6vm86\" (UID: \"5420c28a-0d07-47df-8d12-1ec97ec9707e\") " pod="ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86"
Oct 30 17:18:44.674759 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:44.674731    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5420c28a-0d07-47df-8d12-1ec97ec9707e-tmp-dir\") pod \"test-model-dd5c8f-kserve-554d64c6dc-6vm86\" (UID: \"5420c28a-0d07-47df-8d12-1ec97ec9707e\") " pod="ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86"
Oct 30 17:18:44.675010 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:44.674802    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5420c28a-0d07-47df-8d12-1ec97ec9707e-dshm\") pod \"test-model-dd5c8f-kserve-554d64c6dc-6vm86\" (UID: \"5420c28a-0d07-47df-8d12-1ec97ec9707e\") " pod="ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86"
Oct 30 17:18:44.675010 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:44.674991    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5420c28a-0d07-47df-8d12-1ec97ec9707e-home\") pod \"test-model-dd5c8f-kserve-554d64c6dc-6vm86\" (UID: \"5420c28a-0d07-47df-8d12-1ec97ec9707e\") " pod="ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86"
Oct 30 17:18:44.675313 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:44.675286    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5420c28a-0d07-47df-8d12-1ec97ec9707e-model-cache\") pod \"test-model-dd5c8f-kserve-554d64c6dc-6vm86\" (UID: \"5420c28a-0d07-47df-8d12-1ec97ec9707e\") " pod="ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86"
Oct 30 17:18:44.675506 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:44.675317    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5420c28a-0d07-47df-8d12-1ec97ec9707e-tmp-dir\") pod \"test-model-dd5c8f-kserve-554d64c6dc-6vm86\" (UID: \"5420c28a-0d07-47df-8d12-1ec97ec9707e\") " pod="ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86"
Oct 30 17:18:44.677729 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:44.677702    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5420c28a-0d07-47df-8d12-1ec97ec9707e-tls-certs\") pod \"test-model-dd5c8f-kserve-554d64c6dc-6vm86\" (UID: \"5420c28a-0d07-47df-8d12-1ec97ec9707e\") " pod="ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86"
Oct 30 17:18:44.677930 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:44.677908    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5420c28a-0d07-47df-8d12-1ec97ec9707e-dshm\") pod \"test-model-dd5c8f-kserve-554d64c6dc-6vm86\" (UID: \"5420c28a-0d07-47df-8d12-1ec97ec9707e\") " pod="ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86"
Oct 30 17:18:44.741871 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:44.741704    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrvgf\" (UniqueName: \"kubernetes.io/projected/5420c28a-0d07-47df-8d12-1ec97ec9707e-kube-api-access-xrvgf\") pod \"test-model-dd5c8f-kserve-554d64c6dc-6vm86\" (UID: \"5420c28a-0d07-47df-8d12-1ec97ec9707e\") " pod="ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86"
Oct 30 17:18:44.822430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:44.817757    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86"
Oct 30 17:18:45.204247 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:45.204197    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86"]
Oct 30 17:18:45.384426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:45.374125    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:18:45.408559 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:45.408503    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86" event={"ID":"5420c28a-0d07-47df-8d12-1ec97ec9707e","Type":"ContainerStarted","Data":"50e8c74907c24b2f477d85b057036de768cf4de0331ba962da89c70a01f97886"}
Oct 30 17:18:46.416861 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:46.416822    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86" event={"ID":"5420c28a-0d07-47df-8d12-1ec97ec9707e","Type":"ContainerStarted","Data":"b311e5f9c34e6765bde2101bacc9817e84ec0dc1828ea1f7a61d0d88cea4c3dc"}
Oct 30 17:18:46.464736 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:46.464674    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86" podStartSLOduration=2.464655024 podStartE2EDuration="2.464655024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:18:46.461246853 +0000 UTC m=+1902.896149060" watchObservedRunningTime="2026-10-30 17:18:46.464655024 +0000 UTC m=+1902.899557229"
Oct 30 17:18:46.833552 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:46.833514    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86"
Oct 30 17:18:47.569427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:47.568637    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"]
Oct 30 17:18:47.662446 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:47.652131    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:18:47.662446 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:47.655143    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"]
Oct 30 17:18:47.674882 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:47.674817    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-001a96-openshift-default-dockercfg-cdm4k\""
Oct 30 17:18:47.794953 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:47.794907    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6906457b-d282-4603-8e72-edd173a4afa7-workload-socket\") pod \"e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") " pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:18:47.794953 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:47.794960    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6906457b-d282-4603-8e72-edd173a4afa7-credential-socket\") pod \"e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") " pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:18:47.795235 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:47.795087    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6906457b-d282-4603-8e72-edd173a4afa7-istio-token\") pod \"e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") " pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:18:47.795235 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:47.795141    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6906457b-d282-4603-8e72-edd173a4afa7-istio-envoy\") pod \"e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") " pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:18:47.795353 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:47.795264    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6906457b-d282-4603-8e72-edd173a4afa7-istio-podinfo\") pod \"e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") " pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:18:47.795353 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:47.795321    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6906457b-d282-4603-8e72-edd173a4afa7-workload-certs\") pod \"e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") " pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:18:47.795487 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:47.795392    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6906457b-d282-4603-8e72-edd173a4afa7-istio-data\") pod \"e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") " pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:18:47.795487 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:47.795468    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6906457b-d282-4603-8e72-edd173a4afa7-istiod-ca-cert\") pod \"e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") " pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:18:47.795619 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:47.795602    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqzrf\" (UniqueName: \"kubernetes.io/projected/6906457b-d282-4603-8e72-edd173a4afa7-kube-api-access-rqzrf\") pod \"e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") " pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:18:47.897907 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:47.897803    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqzrf\" (UniqueName: \"kubernetes.io/projected/6906457b-d282-4603-8e72-edd173a4afa7-kube-api-access-rqzrf\") pod \"e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") " pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:18:47.898190 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:47.898162    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6906457b-d282-4603-8e72-edd173a4afa7-workload-socket\") pod \"e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") " pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:18:47.898270 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:47.898193    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6906457b-d282-4603-8e72-edd173a4afa7-credential-socket\") pod \"e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") " pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:18:47.898354 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:47.898337    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6906457b-d282-4603-8e72-edd173a4afa7-istio-token\") pod \"e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") " pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:18:47.898763 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:47.898736    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6906457b-d282-4603-8e72-edd173a4afa7-credential-socket\") pod \"e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") " pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:18:47.899674 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:47.899639    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6906457b-d282-4603-8e72-edd173a4afa7-workload-socket\") pod \"e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") " pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:18:47.901005 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:47.900521    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6906457b-d282-4603-8e72-edd173a4afa7-istio-envoy\") pod \"e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") " pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:18:47.901005 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:47.900617    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6906457b-d282-4603-8e72-edd173a4afa7-istio-podinfo\") pod \"e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") " pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:18:47.901005 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:47.900681    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6906457b-d282-4603-8e72-edd173a4afa7-workload-certs\") pod \"e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") " pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:18:47.901005 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:47.900766    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6906457b-d282-4603-8e72-edd173a4afa7-istio-data\") pod \"e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") " pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:18:47.901005 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:47.900831    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6906457b-d282-4603-8e72-edd173a4afa7-istiod-ca-cert\") pod \"e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") " pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:18:47.901335 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:47.901083    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6906457b-d282-4603-8e72-edd173a4afa7-workload-certs\") pod \"e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") " pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:18:47.901518 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:47.901496    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6906457b-d282-4603-8e72-edd173a4afa7-istio-data\") pod \"e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") " pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:18:47.901678 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:47.901659    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6906457b-d282-4603-8e72-edd173a4afa7-istiod-ca-cert\") pod \"e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") " pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:18:47.927831 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:47.924113    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6906457b-d282-4603-8e72-edd173a4afa7-istio-envoy\") pod \"e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") " pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:18:47.942876 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:47.941016    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqzrf\" (UniqueName: \"kubernetes.io/projected/6906457b-d282-4603-8e72-edd173a4afa7-kube-api-access-rqzrf\") pod \"e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") " pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:18:47.976505 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:47.963938    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6906457b-d282-4603-8e72-edd173a4afa7-istio-token\") pod \"e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") " pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:18:47.976505 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:47.967681    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6906457b-d282-4603-8e72-edd173a4afa7-istio-podinfo\") pod \"e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") " pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:18:48.047428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:48.047066    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:18:49.028256 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:49.028220    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"]
Oct 30 17:18:49.031089 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:49.031032    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:18:49.031278 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:49.031259    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:18:49.031396 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:49.031381    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:18:49.499431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:49.494046    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q" event={"ID":"6906457b-d282-4603-8e72-edd173a4afa7","Type":"ContainerStarted","Data":"478a9cff5355cc0e9b03e6c60c9427195fa3eb2851f17266a5c16756bd379c8a"}
Oct 30 17:18:49.499431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:49.494165    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q" event={"ID":"6906457b-d282-4603-8e72-edd173a4afa7","Type":"ContainerStarted","Data":"dbb30f4d40fc09e4d9605d2e1f83ef9601dbbc3dfbe91b0d0e25fa7776def0a0"}
Oct 30 17:18:49.549259 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:49.549194    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q" podStartSLOduration=2.549173871 podStartE2EDuration="2.549173871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:18:49.5433832 +0000 UTC m=+1905.978285408" watchObservedRunningTime="2026-10-30 17:18:49.549173871 +0000 UTC m=+1905.984076078"
Oct 30 17:18:50.048844 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:50.047690    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:18:50.049600 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:50.049574    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 30 17:18:50.049755 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:50.049734    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q" podUID="6906457b-d282-4603-8e72-edd173a4afa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 30 17:18:51.064936 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:51.060699    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:18:51.508991 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:51.508958    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:18:51.509930 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:51.509910    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:18:54.821277 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:54.821238    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86"
Oct 30 17:18:54.834535 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:54.834507    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86"
Oct 30 17:18:55.583554 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:55.583520    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86"
Oct 30 17:18:56.979190 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:18:56.979149    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:18:58.597750 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:58.597719    4433 generic.go:356] "Generic (PLEG): container finished" podID="13bdf6f8-1211-4221-aa10-480d161dd3f8" containerID="c11e45dd799b0879162a088605376060ff0d44770d15d155b568365d3a6d7fb7" exitCode=0
Oct 30 17:18:58.598310 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:58.597785    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh" event={"ID":"13bdf6f8-1211-4221-aa10-480d161dd3f8","Type":"ContainerDied","Data":"c11e45dd799b0879162a088605376060ff0d44770d15d155b568365d3a6d7fb7"}
Oct 30 17:18:58.598310 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:58.598230    4433 scope.go:122] "RemoveContainer" containerID="c11e45dd799b0879162a088605376060ff0d44770d15d155b568365d3a6d7fb7"
Oct 30 17:18:58.600433 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:58.600380    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:18:58.600530 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:58.600488    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:18:58.600590 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:58.600536    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:18:59.612104 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:18:59.609882    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh" event={"ID":"13bdf6f8-1211-4221-aa10-480d161dd3f8","Type":"ContainerStarted","Data":"aa2cafb4ac7b106ccddc2cc0ad79a94eab52e91d1799f5567885753fc46d52e1"}
Oct 30 17:19:00.030266 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:00.030227    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:19:00.031377 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:00.031342    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 17:19:00.031560 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:00.031394    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh" podUID="13bdf6f8-1211-4221-aa10-480d161dd3f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 17:19:01.030594 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:01.030561    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 17:19:01.031028 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:01.030639    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh" podUID="13bdf6f8-1211-4221-aa10-480d161dd3f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 17:19:02.030840 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:02.030805    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 17:19:02.031243 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:02.030859    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh" podUID="13bdf6f8-1211-4221-aa10-480d161dd3f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 17:19:02.432249 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:02.432214    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:19:03.031143 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:03.031101    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 17:19:03.031629 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:03.031170    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh" podUID="13bdf6f8-1211-4221-aa10-480d161dd3f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 17:19:04.031251 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:04.031214    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 17:19:04.031662 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:04.031280    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh" podUID="13bdf6f8-1211-4221-aa10-480d161dd3f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 17:19:05.030709 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:05.030675    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 17:19:05.030887 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:05.030724    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh" podUID="13bdf6f8-1211-4221-aa10-480d161dd3f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 17:19:06.035870 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:06.035832    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:19:06.037001 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:06.036978    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:19:14.657013 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.656964    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"]
Oct 30 17:19:14.661529 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.661505    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"
Oct 30 17:19:14.682295 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.682265    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2d01ec-openshift-default-dockercfg-g2l4d\""
Oct 30 17:19:14.695851 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.695818    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"]
Oct 30 17:19:14.789647 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.789606    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66crl\" (UniqueName: \"kubernetes.io/projected/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-kube-api-access-66crl\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"
Oct 30 17:19:14.789849 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.789677    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-workload-certs\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"
Oct 30 17:19:14.789849 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.789797    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-istio-data\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"
Oct 30 17:19:14.789849 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.789833    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-istio-token\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"
Oct 30 17:19:14.790001 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.789978    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-istio-podinfo\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"
Oct 30 17:19:14.790054 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.790040    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-credential-socket\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"
Oct 30 17:19:14.790139 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.790121    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-istiod-ca-cert\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"
Oct 30 17:19:14.790373 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.790351    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-workload-socket\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"
Oct 30 17:19:14.790512 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.790455    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-istio-envoy\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"
Oct 30 17:19:14.803917 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.803892    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"]
Oct 30 17:19:14.808037 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.808016    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"
Oct 30 17:19:14.834512 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.834486    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"]
Oct 30 17:19:14.892297 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.892263    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffx8d\" (UniqueName: \"kubernetes.io/projected/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-kube-api-access-ffx8d\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"
Oct 30 17:19:14.892487 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.892364    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-workload-socket\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"
Oct 30 17:19:14.892487 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.892420    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-istio-envoy\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"
Oct 30 17:19:14.892487 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.892465    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-istio-podinfo\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"
Oct 30 17:19:14.892620 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.892492    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-credential-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"
Oct 30 17:19:14.892674 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.892655    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66crl\" (UniqueName: \"kubernetes.io/projected/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-kube-api-access-66crl\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"
Oct 30 17:19:14.892716 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.892697    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-workload-certs\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"
Oct 30 17:19:14.892769 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.892721    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-istio-token\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"
Oct 30 17:19:14.892769 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.892754    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-workload-certs\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"
Oct 30 17:19:14.892848 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.892805    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-workload-socket\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"
Oct 30 17:19:14.892884 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.892858    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-istio-data\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"
Oct 30 17:19:14.892931 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.892880    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-istio-token\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"
Oct 30 17:19:14.892976 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.892958    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-istiod-ca-cert\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"
Oct 30 17:19:14.893026 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.893005    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-workload-certs\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"
Oct 30 17:19:14.893101 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.893065    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-istio-podinfo\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"
Oct 30 17:19:14.893144 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.893121    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-istio-data\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"
Oct 30 17:19:14.893197 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.893147    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-credential-socket\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"
Oct 30 17:19:14.893197 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.893186    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-istiod-ca-cert\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"
Oct 30 17:19:14.893275 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.893213    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-istio-data\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"
Oct 30 17:19:14.893313 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.893277    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-workload-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"
Oct 30 17:19:14.893313 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.893306    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-istio-envoy\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"
Oct 30 17:19:14.893452 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.893423    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-credential-socket\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"
Oct 30 17:19:14.893849 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.893828    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-istiod-ca-cert\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"
Oct 30 17:19:14.894949 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.894926    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-istio-envoy\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"
Oct 30 17:19:14.895505 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.895487    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-istio-podinfo\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"
Oct 30 17:19:14.915692 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.915632    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66crl\" (UniqueName: \"kubernetes.io/projected/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-kube-api-access-66crl\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"
Oct 30 17:19:14.916323 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.916302    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-istio-token\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"
Oct 30 17:19:14.975542 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.975507    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"
Oct 30 17:19:14.994576 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.994547    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffx8d\" (UniqueName: \"kubernetes.io/projected/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-kube-api-access-ffx8d\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"
Oct 30 17:19:14.994702 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.994674    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-istio-podinfo\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"
Oct 30 17:19:14.994759 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.994702    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-credential-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"
Oct 30 17:19:14.994852 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.994822    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-istio-token\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"
Oct 30 17:19:14.994919 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.994902    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-workload-certs\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"
Oct 30 17:19:14.995017 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.995003    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-istiod-ca-cert\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"
Oct 30 17:19:14.995126 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.995107    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-credential-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"
Oct 30 17:19:14.995126 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.995119    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-istio-data\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"
Oct 30 17:19:14.995250 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.995191    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-workload-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"
Oct 30 17:19:14.995250 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.995219    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-istio-envoy\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"
Oct 30 17:19:14.995250 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.995228    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-workload-certs\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"
Oct 30 17:19:14.995531 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.995509    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-workload-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"
Oct 30 17:19:14.995631 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.995605    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-istio-data\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"
Oct 30 17:19:14.995818 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.995795    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-istiod-ca-cert\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"
Oct 30 17:19:14.997472 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.997451    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-istio-envoy\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"
Oct 30 17:19:14.997655 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:14.997638    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-istio-podinfo\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"
Oct 30 17:19:15.017666 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:15.017636    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-istio-token\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"
Oct 30 17:19:15.017847 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:15.017825    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffx8d\" (UniqueName: \"kubernetes.io/projected/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-kube-api-access-ffx8d\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"
Oct 30 17:19:15.123016 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:15.122981    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"
Oct 30 17:19:15.298989 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:15.298957    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"]
Oct 30 17:19:15.300889 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:19:15.300862    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeead3a6a_f87a_43c0_94fd_e2c0ced708ba.slice/crio-5e03f5fb9f3f612866f7a8117a13b4a43df4c3c0982da5a37345aa0d33870680 WatchSource:0}: Error finding container 5e03f5fb9f3f612866f7a8117a13b4a43df4c3c0982da5a37345aa0d33870680: Status 404 returned error can't find the container with id 5e03f5fb9f3f612866f7a8117a13b4a43df4c3c0982da5a37345aa0d33870680
Oct 30 17:19:15.303743 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:15.303710    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:19:15.303839 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:15.303775    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:19:15.303839 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:15.303803    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:19:15.429863 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:15.429832    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"]
Oct 30 17:19:15.432375 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:19:15.432339    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103bc667_ffe4_4cc3_a66d_fc85ed8a0a51.slice/crio-c1fa43edc92f8850c9d9f3c46fd16345ce64beb7851b79c0a500f71623094283 WatchSource:0}: Error finding container c1fa43edc92f8850c9d9f3c46fd16345ce64beb7851b79c0a500f71623094283: Status 404 returned error can't find the container with id c1fa43edc92f8850c9d9f3c46fd16345ce64beb7851b79c0a500f71623094283
Oct 30 17:19:15.435874 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:15.435829    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:19:15.435989 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:15.435925    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:19:15.435989 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:15.435970    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:19:15.758431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:15.758328    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw" event={"ID":"eead3a6a-f87a-43c0-94fd-e2c0ced708ba","Type":"ContainerStarted","Data":"259a116c386cf1fc18846376120dcb247ad5a50fd624e12dc12324ead32b6c5f"}
Oct 30 17:19:15.758431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:15.758396    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw" event={"ID":"eead3a6a-f87a-43c0-94fd-e2c0ced708ba","Type":"ContainerStarted","Data":"5e03f5fb9f3f612866f7a8117a13b4a43df4c3c0982da5a37345aa0d33870680"}
Oct 30 17:19:15.760392 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:15.760360    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv" event={"ID":"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51","Type":"ContainerStarted","Data":"a913ca1e0159a6ef5bd341ff843691ed2efb9e8ca38e53cdb110e9d4eafb3a9c"}
Oct 30 17:19:15.760549 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:15.760457    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv" event={"ID":"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51","Type":"ContainerStarted","Data":"c1fa43edc92f8850c9d9f3c46fd16345ce64beb7851b79c0a500f71623094283"}
Oct 30 17:19:15.808862 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:15.808791    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw" podStartSLOduration=1.8087707769999999 podStartE2EDuration="1.808770777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:19:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:19:15.803011614 +0000 UTC m=+1932.237913844" watchObservedRunningTime="2026-10-30 17:19:15.808770777 +0000 UTC m=+1932.243672984"
Oct 30 17:19:15.851210 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:15.851151    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv" podStartSLOduration=1.851132363 podStartE2EDuration="1.851132363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:19:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:19:15.847258781 +0000 UTC m=+1932.282160988" watchObservedRunningTime="2026-10-30 17:19:15.851132363 +0000 UTC m=+1932.286034571"
Oct 30 17:19:15.977620 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:15.977532    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"
Oct 30 17:19:16.123551 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:16.123467    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"
Oct 30 17:19:16.125442 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:16.125378    4433 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 17:19:16.125589 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:16.125461    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv" podUID="103bc667-ffe4-4cc3-a66d-fc85ed8a0a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 17:19:17.002756 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:17.002713    4433 patch_prober.go:29] interesting pod/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 17:19:17.003224 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:17.002815    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw" podUID="eead3a6a-f87a-43c0-94fd-e2c0ced708ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 17:19:17.125673 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:17.123651    4433 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 17:19:17.125673 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:17.123704    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv" podUID="103bc667-ffe4-4cc3-a66d-fc85ed8a0a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 17:19:17.982445 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:17.977839    4433 patch_prober.go:29] interesting pod/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 17:19:17.982445 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:17.977975    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw" podUID="eead3a6a-f87a-43c0-94fd-e2c0ced708ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 17:19:18.124454 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:18.123826    4433 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 17:19:18.124454 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:18.123900    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv" podUID="103bc667-ffe4-4cc3-a66d-fc85ed8a0a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 17:19:18.978430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:18.977828    4433 patch_prober.go:29] interesting pod/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 17:19:18.978430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:18.977892    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw" podUID="eead3a6a-f87a-43c0-94fd-e2c0ced708ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 17:19:19.128978 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:19.128939    4433 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 17:19:19.129651 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:19.129609    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv" podUID="103bc667-ffe4-4cc3-a66d-fc85ed8a0a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 17:19:19.976905 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:19.976859    4433 patch_prober.go:29] interesting pod/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 17:19:19.977214 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:19.977180    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw" podUID="eead3a6a-f87a-43c0-94fd-e2c0ced708ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 17:19:20.124050 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:20.124013    4433 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 17:19:20.124224 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:20.124068    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv" podUID="103bc667-ffe4-4cc3-a66d-fc85ed8a0a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 17:19:20.985541 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:20.985503    4433 patch_prober.go:29] interesting pod/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 17:19:20.986258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:20.986226    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw" podUID="eead3a6a-f87a-43c0-94fd-e2c0ced708ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 17:19:21.123759 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:21.123727    4433 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 17:19:21.123996 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:21.123976    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv" podUID="103bc667-ffe4-4cc3-a66d-fc85ed8a0a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 17:19:21.976513 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:21.976478    4433 patch_prober.go:29] interesting pod/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 17:19:21.976747 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:21.976536    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw" podUID="eead3a6a-f87a-43c0-94fd-e2c0ced708ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 17:19:22.124230 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:22.124193    4433 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 17:19:22.124685 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:22.124261    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv" podUID="103bc667-ffe4-4cc3-a66d-fc85ed8a0a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 17:19:22.982056 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:22.982022    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"
Oct 30 17:19:22.982505 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:22.982483    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"
Oct 30 17:19:22.983168 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:22.983150    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"
Oct 30 17:19:23.131160 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:23.131124    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"
Oct 30 17:19:23.132818 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:23.132798    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"
Oct 30 17:19:23.133577 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:19:23.133556    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"
Oct 30 17:19:26.979189 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:19:26.979151    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:19:56.978969 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:19:56.978930    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:20:26.979458 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:20:26.979418    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:20:35.507226 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:35.507186    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"]
Oct 30 17:20:35.507794 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:35.507442    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2" podUID="df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3" containerName="istio-proxy" containerID="cri-o://d01ac2962194a9324768421c0b8f400f64402951bf5594094e29126112894300" gracePeriod=30
Oct 30 17:20:35.519177 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:35.519141    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"]
Oct 30 17:20:35.519398 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:35.519354    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc" podUID="781b1fab-ca9f-4df7-8061-065561b69193" containerName="istio-proxy" containerID="cri-o://3b7eec95adcfffdad27c9468aefa24a61224b90e3792eb2b4a63fa77039faabe" gracePeriod=30
Oct 30 17:20:35.522698 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:35.522670    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"]
Oct 30 17:20:35.522906 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:35.522870    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk" podUID="72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc" containerName="istio-proxy" containerID="cri-o://59c124e9845acc942314f6848c269ad97b968f3f02757fef8ac95366dd3e6220" gracePeriod=30
Oct 30 17:20:40.577854 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.577817    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:20:40.932361 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.932334    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"
Oct 30 17:20:40.934994 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.934975    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"
Oct 30 17:20:40.938787 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.938769    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"
Oct 30 17:20:40.983859 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.983832    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-kube-api-access-2j7xp\" (UniqueName: \"kubernetes.io/projected/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-kube-api-access-2j7xp\") pod \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") "
Oct 30 17:20:40.984055 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.983908    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-istio-token\" (UniqueName: \"kubernetes.io/projected/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-istio-token\") pod \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") "
Oct 30 17:20:40.984055 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.983937    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-istio-podinfo\") pod \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") "
Oct 30 17:20:40.984055 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.983999    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-kube-api-access-8pxhl\" (UniqueName: \"kubernetes.io/projected/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-kube-api-access-8pxhl\") pod \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") "
Oct 30 17:20:40.984368 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.984337    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/781b1fab-ca9f-4df7-8061-065561b69193-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/781b1fab-ca9f-4df7-8061-065561b69193-istio-podinfo\") pod \"781b1fab-ca9f-4df7-8061-065561b69193\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") "
Oct 30 17:20:40.984558 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.984386    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-credential-socket\") pod \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") "
Oct 30 17:20:40.984558 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.984502    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/781b1fab-ca9f-4df7-8061-065561b69193-istio-token\" (UniqueName: \"kubernetes.io/projected/781b1fab-ca9f-4df7-8061-065561b69193-istio-token\") pod \"781b1fab-ca9f-4df7-8061-065561b69193\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") "
Oct 30 17:20:40.984675 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.984634    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-workload-certs\") pod \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") "
Oct 30 17:20:40.984736 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.984674    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/781b1fab-ca9f-4df7-8061-065561b69193-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/781b1fab-ca9f-4df7-8061-065561b69193-workload-socket\") pod \"781b1fab-ca9f-4df7-8061-065561b69193\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") "
Oct 30 17:20:40.984736 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.984675    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-credential-socket" pod "df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3" (UID: "df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:20:40.984736 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.984705    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-istio-data\") pod \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") "
Oct 30 17:20:40.984906 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.984753    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-istiod-ca-cert\") pod \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") "
Oct 30 17:20:40.984906 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.984788    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/781b1fab-ca9f-4df7-8061-065561b69193-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/781b1fab-ca9f-4df7-8061-065561b69193-workload-certs\") pod \"781b1fab-ca9f-4df7-8061-065561b69193\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") "
Oct 30 17:20:40.984906 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.984814    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-istio-token\" (UniqueName: \"kubernetes.io/projected/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-istio-token\") pod \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") "
Oct 30 17:20:40.984906 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.984841    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-workload-certs\") pod \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") "
Oct 30 17:20:40.984906 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.984887    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-workload-certs" pod "72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc" (UID: "72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:20:40.984906 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.984899    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/781b1fab-ca9f-4df7-8061-065561b69193-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/781b1fab-ca9f-4df7-8061-065561b69193-istio-envoy\") pod \"781b1fab-ca9f-4df7-8061-065561b69193\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") "
Oct 30 17:20:40.986346 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.984957    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/781b1fab-ca9f-4df7-8061-065561b69193-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/781b1fab-ca9f-4df7-8061-065561b69193-istiod-ca-cert\") pod \"781b1fab-ca9f-4df7-8061-065561b69193\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") "
Oct 30 17:20:40.986346 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.985003    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-istio-envoy\") pod \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") "
Oct 30 17:20:40.986346 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.985091    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-istio-podinfo\") pod \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") "
Oct 30 17:20:40.986346 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.985132    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/781b1fab-ca9f-4df7-8061-065561b69193-kube-api-access-dcmth\" (UniqueName: \"kubernetes.io/projected/781b1fab-ca9f-4df7-8061-065561b69193-kube-api-access-dcmth\") pod \"781b1fab-ca9f-4df7-8061-065561b69193\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") "
Oct 30 17:20:40.986346 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.985168    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-workload-socket\") pod \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") "
Oct 30 17:20:40.986346 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.985204    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-istio-envoy\") pod \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") "
Oct 30 17:20:40.986346 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.985322    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-credential-socket\") pod \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") "
Oct 30 17:20:40.986346 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.985429    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/781b1fab-ca9f-4df7-8061-065561b69193-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/781b1fab-ca9f-4df7-8061-065561b69193-credential-socket\") pod \"781b1fab-ca9f-4df7-8061-065561b69193\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") "
Oct 30 17:20:40.986346 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.985443    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-workload-certs" pod "df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3" (UID: "df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:20:40.986346 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.985526    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-istiod-ca-cert\") pod \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") "
Oct 30 17:20:40.986346 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.985560    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/781b1fab-ca9f-4df7-8061-065561b69193-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/781b1fab-ca9f-4df7-8061-065561b69193-istio-data\") pod \"781b1fab-ca9f-4df7-8061-065561b69193\" (UID: \"781b1fab-ca9f-4df7-8061-065561b69193\") "
Oct 30 17:20:40.986346 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.985591    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-workload-socket\") pod \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\" (UID: \"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3\") "
Oct 30 17:20:40.986346 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.985619    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-istio-data\") pod \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\" (UID: \"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc\") "
Oct 30 17:20:40.986346 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.985642    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781b1fab-ca9f-4df7-8061-065561b69193-workload-socket" pod "781b1fab-ca9f-4df7-8061-065561b69193" (UID: "781b1fab-ca9f-4df7-8061-065561b69193"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:20:40.986346 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.985824    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-istio-data" pod "df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3" (UID: "df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:20:40.986346 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.986028    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-istio-data" pod "72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc" (UID: "72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:20:40.988194 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.986564    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:20:40.988194 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.986588    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:20:40.988194 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.986603    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/781b1fab-ca9f-4df7-8061-065561b69193-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:20:40.988194 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.986615    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:20:40.988194 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.986631    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:20:40.988194 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.986644    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:20:40.988194 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.986869    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/781b1fab-ca9f-4df7-8061-065561b69193-istio-podinfo" pod "781b1fab-ca9f-4df7-8061-065561b69193" (UID: "781b1fab-ca9f-4df7-8061-065561b69193"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:20:40.988194 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.987044    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-istiod-ca-cert" pod "72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc" (UID: "72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:20:40.988194 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.987166    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781b1fab-ca9f-4df7-8061-065561b69193-workload-certs" pod "781b1fab-ca9f-4df7-8061-065561b69193" (UID: "781b1fab-ca9f-4df7-8061-065561b69193"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:20:40.988914 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.988889    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781b1fab-ca9f-4df7-8061-065561b69193-istio-envoy" pod "781b1fab-ca9f-4df7-8061-065561b69193" (UID: "781b1fab-ca9f-4df7-8061-065561b69193"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:20:40.989130 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.989101    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-istio-podinfo" pod "df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3" (UID: "df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:20:40.989248 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.989231    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-kube-api-access-8pxhl" pod "72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc" (UID: "72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc"). InnerVolumeSpecName "kube-api-access-8pxhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:20:40.989306 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.989260    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/781b1fab-ca9f-4df7-8061-065561b69193-istio-token" pod "781b1fab-ca9f-4df7-8061-065561b69193" (UID: "781b1fab-ca9f-4df7-8061-065561b69193"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:20:40.989478 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.989449    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-workload-socket" pod "72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc" (UID: "72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:20:40.989607 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.989561    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-credential-socket" pod "72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc" (UID: "72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:20:40.989696 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.989645    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/781b1fab-ca9f-4df7-8061-065561b69193-kube-api-access-dcmth" pod "781b1fab-ca9f-4df7-8061-065561b69193" (UID: "781b1fab-ca9f-4df7-8061-065561b69193"). InnerVolumeSpecName "kube-api-access-dcmth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:20:40.989739 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.989649    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/781b1fab-ca9f-4df7-8061-065561b69193-istiod-ca-cert" pod "781b1fab-ca9f-4df7-8061-065561b69193" (UID: "781b1fab-ca9f-4df7-8061-065561b69193"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:20:40.989797 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.989776    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781b1fab-ca9f-4df7-8061-065561b69193-istio-data" pod "781b1fab-ca9f-4df7-8061-065561b69193" (UID: "781b1fab-ca9f-4df7-8061-065561b69193"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:20:40.989915 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.989898    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781b1fab-ca9f-4df7-8061-065561b69193-credential-socket" pod "781b1fab-ca9f-4df7-8061-065561b69193" (UID: "781b1fab-ca9f-4df7-8061-065561b69193"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:20:40.989968 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.989919    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-workload-socket" pod "df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3" (UID: "df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:20:40.990013 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.989962    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-istiod-ca-cert" pod "df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3" (UID: "df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:20:40.991339 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.991318    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-istio-envoy" pod "df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3" (UID: "df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:20:40.991594 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.991575    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-istio-podinfo" pod "72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc" (UID: "72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:20:40.991681 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.991588    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-istio-envoy" pod "72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc" (UID: "72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:20:40.991681 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.991655    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-kube-api-access-2j7xp" pod "df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3" (UID: "df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3"). InnerVolumeSpecName "kube-api-access-2j7xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:20:40.991814 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.991792    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-istio-token" pod "72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc" (UID: "72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:20:40.993081 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:40.993057    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-istio-token" pod "df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3" (UID: "df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:20:41.088330 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.088290    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/781b1fab-ca9f-4df7-8061-065561b69193-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:20:41.088330 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.088327    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/781b1fab-ca9f-4df7-8061-065561b69193-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:20:41.088599 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.088340    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:20:41.088599 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.088352    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:20:41.088599 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.088364    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcmth\" (UniqueName: \"kubernetes.io/projected/781b1fab-ca9f-4df7-8061-065561b69193-kube-api-access-dcmth\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:20:41.088599 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.088376    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:20:41.088599 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.088387    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:20:41.088599 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.088399    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:20:41.088599 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.088430    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/781b1fab-ca9f-4df7-8061-065561b69193-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:20:41.088599 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.088443    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:20:41.088599 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.088455    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/781b1fab-ca9f-4df7-8061-065561b69193-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:20:41.088599 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.088467    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:20:41.088599 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.088480    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j7xp\" (UniqueName: \"kubernetes.io/projected/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-kube-api-access-2j7xp\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:20:41.088599 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.088491    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:20:41.088599 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.088505    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:20:41.088599 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.088516    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pxhl\" (UniqueName: \"kubernetes.io/projected/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-kube-api-access-8pxhl\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:20:41.088599 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.088529    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/781b1fab-ca9f-4df7-8061-065561b69193-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:20:41.088599 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.088541    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/781b1fab-ca9f-4df7-8061-065561b69193-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:20:41.088599 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.088552    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:20:41.088599 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.088566    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/781b1fab-ca9f-4df7-8061-065561b69193-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:20:41.088599 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.088579    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:20:41.453224 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.453187    4433 generic.go:356] "Generic (PLEG): container finished" podID="72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc" containerID="59c124e9845acc942314f6848c269ad97b968f3f02757fef8ac95366dd3e6220" exitCode=0
Oct 30 17:20:41.453436 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.453239    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"
Oct 30 17:20:41.453436 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.453269    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk" event={"ID":"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc","Type":"ContainerDied","Data":"59c124e9845acc942314f6848c269ad97b968f3f02757fef8ac95366dd3e6220"}
Oct 30 17:20:41.453436 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.453328    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk" event={"ID":"72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc","Type":"ContainerDied","Data":"f922764e83698445a3a662630edc4a01d74fdd9c39e5e42966f120a7eea59c63"}
Oct 30 17:20:41.453436 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.453353    4433 scope.go:122] "RemoveContainer" containerID="59c124e9845acc942314f6848c269ad97b968f3f02757fef8ac95366dd3e6220"
Oct 30 17:20:41.454943 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.454918    4433 generic.go:356] "Generic (PLEG): container finished" podID="df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3" containerID="d01ac2962194a9324768421c0b8f400f64402951bf5594094e29126112894300" exitCode=0
Oct 30 17:20:41.455060 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.454980    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"
Oct 30 17:20:41.455060 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.455025    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2" event={"ID":"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3","Type":"ContainerDied","Data":"d01ac2962194a9324768421c0b8f400f64402951bf5594094e29126112894300"}
Oct 30 17:20:41.455161 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.455077    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2" event={"ID":"df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3","Type":"ContainerDied","Data":"8977016f57cccfc42e452ef4fdb74b1b9ff68992c277b9c60fa2852ff43bd9c5"}
Oct 30 17:20:41.464953 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.464925    4433 generic.go:356] "Generic (PLEG): container finished" podID="781b1fab-ca9f-4df7-8061-065561b69193" containerID="3b7eec95adcfffdad27c9468aefa24a61224b90e3792eb2b4a63fa77039faabe" exitCode=0
Oct 30 17:20:41.465103 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.465083    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc" event={"ID":"781b1fab-ca9f-4df7-8061-065561b69193","Type":"ContainerDied","Data":"3b7eec95adcfffdad27c9468aefa24a61224b90e3792eb2b4a63fa77039faabe"}
Oct 30 17:20:41.465163 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.465139    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc" event={"ID":"781b1fab-ca9f-4df7-8061-065561b69193","Type":"ContainerDied","Data":"6f34e11b59e01c2a9ba057a239d8139ee86d86d93bca31e7f5b120618ce0cae5"}
Oct 30 17:20:41.465234 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.465220    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"
Oct 30 17:20:41.465938 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.465914    4433 scope.go:122] "RemoveContainer" containerID="59c124e9845acc942314f6848c269ad97b968f3f02757fef8ac95366dd3e6220"
Oct 30 17:20:41.466248 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:20:41.466214    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59c124e9845acc942314f6848c269ad97b968f3f02757fef8ac95366dd3e6220\": container with ID starting with 59c124e9845acc942314f6848c269ad97b968f3f02757fef8ac95366dd3e6220 not found: ID does not exist" containerID="59c124e9845acc942314f6848c269ad97b968f3f02757fef8ac95366dd3e6220"
Oct 30 17:20:41.466353 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.466247    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59c124e9845acc942314f6848c269ad97b968f3f02757fef8ac95366dd3e6220"} err="failed to get container status \"59c124e9845acc942314f6848c269ad97b968f3f02757fef8ac95366dd3e6220\": rpc error: code = NotFound desc = could not find container \"59c124e9845acc942314f6848c269ad97b968f3f02757fef8ac95366dd3e6220\": container with ID starting with 59c124e9845acc942314f6848c269ad97b968f3f02757fef8ac95366dd3e6220 not found: ID does not exist"
Oct 30 17:20:41.466353 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.466270    4433 scope.go:122] "RemoveContainer" containerID="d01ac2962194a9324768421c0b8f400f64402951bf5594094e29126112894300"
Oct 30 17:20:41.476578 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.476559    4433 scope.go:122] "RemoveContainer" containerID="d01ac2962194a9324768421c0b8f400f64402951bf5594094e29126112894300"
Oct 30 17:20:41.476843 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:20:41.476823    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d01ac2962194a9324768421c0b8f400f64402951bf5594094e29126112894300\": container with ID starting with d01ac2962194a9324768421c0b8f400f64402951bf5594094e29126112894300 not found: ID does not exist" containerID="d01ac2962194a9324768421c0b8f400f64402951bf5594094e29126112894300"
Oct 30 17:20:41.476938 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.476855    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d01ac2962194a9324768421c0b8f400f64402951bf5594094e29126112894300"} err="failed to get container status \"d01ac2962194a9324768421c0b8f400f64402951bf5594094e29126112894300\": rpc error: code = NotFound desc = could not find container \"d01ac2962194a9324768421c0b8f400f64402951bf5594094e29126112894300\": container with ID starting with d01ac2962194a9324768421c0b8f400f64402951bf5594094e29126112894300 not found: ID does not exist"
Oct 30 17:20:41.476938 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.476877    4433 scope.go:122] "RemoveContainer" containerID="3b7eec95adcfffdad27c9468aefa24a61224b90e3792eb2b4a63fa77039faabe"
Oct 30 17:20:41.486303 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.486286    4433 scope.go:122] "RemoveContainer" containerID="3b7eec95adcfffdad27c9468aefa24a61224b90e3792eb2b4a63fa77039faabe"
Oct 30 17:20:41.486576 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:20:41.486557    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b7eec95adcfffdad27c9468aefa24a61224b90e3792eb2b4a63fa77039faabe\": container with ID starting with 3b7eec95adcfffdad27c9468aefa24a61224b90e3792eb2b4a63fa77039faabe not found: ID does not exist" containerID="3b7eec95adcfffdad27c9468aefa24a61224b90e3792eb2b4a63fa77039faabe"
Oct 30 17:20:41.486639 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.486582    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b7eec95adcfffdad27c9468aefa24a61224b90e3792eb2b4a63fa77039faabe"} err="failed to get container status \"3b7eec95adcfffdad27c9468aefa24a61224b90e3792eb2b4a63fa77039faabe\": rpc error: code = NotFound desc = could not find container \"3b7eec95adcfffdad27c9468aefa24a61224b90e3792eb2b4a63fa77039faabe\": container with ID starting with 3b7eec95adcfffdad27c9468aefa24a61224b90e3792eb2b4a63fa77039faabe not found: ID does not exist"
Oct 30 17:20:41.538021 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.537982    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"]
Oct 30 17:20:41.549954 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.549921    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-wmkx2"]
Oct 30 17:20:41.579738 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.579691    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"]
Oct 30 17:20:41.585041 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.585009    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8vzdk"]
Oct 30 17:20:41.632226 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.631448    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"]
Oct 30 17:20:41.637397 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.637370    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zlfpc"]
Oct 30 17:20:41.645330 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.645298    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7"]
Oct 30 17:20:41.705616 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.705539    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc" path="/var/lib/kubelet/pods/72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc/volumes"
Oct 30 17:20:41.706066 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.706051    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="781b1fab-ca9f-4df7-8061-065561b69193" path="/var/lib/kubelet/pods/781b1fab-ca9f-4df7-8061-065561b69193/volumes"
Oct 30 17:20:41.706578 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:41.706562    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3" path="/var/lib/kubelet/pods/df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3/volumes"
Oct 30 17:20:46.737195 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:46.737111    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh"]
Oct 30 17:20:46.737857 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:46.737838    4433 cpu_manager.go:414] "containerMap: removing container" podUID="df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3" containerName="istio-proxy"
Oct 30 17:20:46.737936 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:46.737858    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3" containerName="istio-proxy"
Oct 30 17:20:46.737936 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:46.737896    4433 cpu_manager.go:414] "containerMap: removing container" podUID="781b1fab-ca9f-4df7-8061-065561b69193" containerName="istio-proxy"
Oct 30 17:20:46.737936 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:46.737901    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="781b1fab-ca9f-4df7-8061-065561b69193" containerName="istio-proxy"
Oct 30 17:20:46.737936 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:46.737918    4433 cpu_manager.go:414] "containerMap: removing container" podUID="72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc" containerName="istio-proxy"
Oct 30 17:20:46.737936 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:46.737923    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc" containerName="istio-proxy"
Oct 30 17:20:46.738203 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:46.738016    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="72cf77f0-bc7d-4bc1-a655-f4b98a34f3cc" containerName="istio-proxy"
Oct 30 17:20:46.738203 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:46.738029    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="df3ba2d5-bc3c-48dc-a8fe-b3d850ea2eb3" containerName="istio-proxy"
Oct 30 17:20:46.738203 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:46.738039    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="781b1fab-ca9f-4df7-8061-065561b69193" containerName="istio-proxy"
Oct 30 17:20:46.743306 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:46.743282    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh"
Oct 30 17:20:46.763710 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:46.763681    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c45877\"/\"kube-root-ca.crt\""
Oct 30 17:20:46.773491 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:46.773449    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-tls-certs\") pod \"test-model-c45877-kserve-7bfd8f46f5-zx7fh\" (UID: \"9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f\") " pod="ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh"
Oct 30 17:20:46.773828 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:46.773808    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-tmp-dir\") pod \"test-model-c45877-kserve-7bfd8f46f5-zx7fh\" (UID: \"9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f\") " pod="ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh"
Oct 30 17:20:46.774026 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:46.774006    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f79kf\" (UniqueName: \"kubernetes.io/projected/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-kube-api-access-f79kf\") pod \"test-model-c45877-kserve-7bfd8f46f5-zx7fh\" (UID: \"9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f\") " pod="ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh"
Oct 30 17:20:46.774199 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:46.774182    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-model-cache\") pod \"test-model-c45877-kserve-7bfd8f46f5-zx7fh\" (UID: \"9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f\") " pod="ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh"
Oct 30 17:20:46.774469 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:46.774450    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-home\") pod \"test-model-c45877-kserve-7bfd8f46f5-zx7fh\" (UID: \"9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f\") " pod="ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh"
Oct 30 17:20:46.775284 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:46.775262    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-dshm\") pod \"test-model-c45877-kserve-7bfd8f46f5-zx7fh\" (UID: \"9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f\") " pod="ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh"
Oct 30 17:20:46.786427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:46.781914    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c45877\"/\"test-model-c45877-kserve-self-signed-certs\""
Oct 30 17:20:46.786427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:46.781994    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c45877\"/\"openshift-service-ca.crt\""
Oct 30 17:20:46.786427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:46.782691    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c45877\"/\"default-dockercfg-b9dv7\""
Oct 30 17:20:46.790620 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:46.790592    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh"]
Oct 30 17:20:46.888467 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:46.886349    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-home\") pod \"test-model-c45877-kserve-7bfd8f46f5-zx7fh\" (UID: \"9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f\") " pod="ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh"
Oct 30 17:20:46.888467 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:46.886583    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-dshm\") pod \"test-model-c45877-kserve-7bfd8f46f5-zx7fh\" (UID: \"9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f\") " pod="ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh"
Oct 30 17:20:46.888467 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:46.886648    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-tls-certs\") pod \"test-model-c45877-kserve-7bfd8f46f5-zx7fh\" (UID: \"9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f\") " pod="ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh"
Oct 30 17:20:46.888467 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:46.886716    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-tmp-dir\") pod \"test-model-c45877-kserve-7bfd8f46f5-zx7fh\" (UID: \"9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f\") " pod="ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh"
Oct 30 17:20:46.888467 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:46.886765    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f79kf\" (UniqueName: \"kubernetes.io/projected/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-kube-api-access-f79kf\") pod \"test-model-c45877-kserve-7bfd8f46f5-zx7fh\" (UID: \"9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f\") " pod="ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh"
Oct 30 17:20:46.888467 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:46.886797    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-model-cache\") pod \"test-model-c45877-kserve-7bfd8f46f5-zx7fh\" (UID: \"9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f\") " pod="ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh"
Oct 30 17:20:46.888467 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:46.887272    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-model-cache\") pod \"test-model-c45877-kserve-7bfd8f46f5-zx7fh\" (UID: \"9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f\") " pod="ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh"
Oct 30 17:20:46.888467 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:46.888388    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-home\") pod \"test-model-c45877-kserve-7bfd8f46f5-zx7fh\" (UID: \"9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f\") " pod="ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh"
Oct 30 17:20:46.889918 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:46.889481    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-tmp-dir\") pod \"test-model-c45877-kserve-7bfd8f46f5-zx7fh\" (UID: \"9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f\") " pod="ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh"
Oct 30 17:20:46.891147 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:46.891118    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-dshm\") pod \"test-model-c45877-kserve-7bfd8f46f5-zx7fh\" (UID: \"9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f\") " pod="ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh"
Oct 30 17:20:46.895144 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:46.894639    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-tls-certs\") pod \"test-model-c45877-kserve-7bfd8f46f5-zx7fh\" (UID: \"9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f\") " pod="ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh"
Oct 30 17:20:46.911922 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:46.911883    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f79kf\" (UniqueName: \"kubernetes.io/projected/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-kube-api-access-f79kf\") pod \"test-model-c45877-kserve-7bfd8f46f5-zx7fh\" (UID: \"9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f\") " pod="ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh"
Oct 30 17:20:47.092472 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:47.091702    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh"
Oct 30 17:20:47.656669 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:47.655697    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh"]
Oct 30 17:20:47.821424 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:47.820764    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh" event={"ID":"9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f","Type":"ContainerStarted","Data":"eda6c7e1c95c50c8ff4d91944235528eef01c7af507b789845bb20b4612a37e4"}
Oct 30 17:20:47.821424 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:47.820817    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh" event={"ID":"9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f","Type":"ContainerStarted","Data":"2f8603eeb29eaeab66c4eb146957d1655367fddb2936e75178bd7b1a98238223"}
Oct 30 17:20:47.897615 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:47.897532    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh" podStartSLOduration=1.897512168 podStartE2EDuration="1.897512168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:20:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:20:47.872382224 +0000 UTC m=+2024.307284432" watchObservedRunningTime="2026-10-30 17:20:47.897512168 +0000 UTC m=+2024.332414377"
Oct 30 17:20:48.096478 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:48.094480    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh"
Oct 30 17:20:52.550568 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:52.550534    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:20:52.550995 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:52.550594    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q" podUID="6906457b-d282-4603-8e72-edd173a4afa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:20:53.904936 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:53.904894    4433 generic.go:356] "Generic (PLEG): container finished" podID="6906457b-d282-4603-8e72-edd173a4afa7" containerID="478a9cff5355cc0e9b03e6c60c9427195fa3eb2851f17266a5c16756bd379c8a" exitCode=0
Oct 30 17:20:53.905505 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:53.905004    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q" event={"ID":"6906457b-d282-4603-8e72-edd173a4afa7","Type":"ContainerDied","Data":"478a9cff5355cc0e9b03e6c60c9427195fa3eb2851f17266a5c16756bd379c8a"}
Oct 30 17:20:53.905627 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:53.905606    4433 scope.go:122] "RemoveContainer" containerID="478a9cff5355cc0e9b03e6c60c9427195fa3eb2851f17266a5c16756bd379c8a"
Oct 30 17:20:53.911247 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:53.911182    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:20:53.911396 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:53.911290    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:20:53.911396 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:53.911334    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:20:54.913678 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:54.913642    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q" event={"ID":"6906457b-d282-4603-8e72-edd173a4afa7","Type":"ContainerStarted","Data":"549a6c08fedba5e518e75ca00986b095623032ba88e99849abc50fc4d6ff2aaa"}
Oct 30 17:20:55.047495 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:55.047446    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:20:55.047721 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:55.047647    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 30 17:20:55.047721 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:55.047709    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q" podUID="6906457b-d282-4603-8e72-edd173a4afa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 30 17:20:56.047550 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:56.047511    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 30 17:20:56.048054 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:56.047577    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q" podUID="6906457b-d282-4603-8e72-edd173a4afa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 30 17:20:56.699469 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:56.699377    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7" podUID="4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe" containerName="main" containerID="cri-o://0ceb38e3f33ceff1771d1c8e56d54334b13322f739818972d787e197c5d98c46" gracePeriod=45
Oct 30 17:20:56.935422 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:56.935372    4433 generic.go:356] "Generic (PLEG): container finished" podID="4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe" containerID="0ceb38e3f33ceff1771d1c8e56d54334b13322f739818972d787e197c5d98c46" exitCode=0
Oct 30 17:20:56.935597 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:56.935446    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7" event={"ID":"4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe","Type":"ContainerDied","Data":"0ceb38e3f33ceff1771d1c8e56d54334b13322f739818972d787e197c5d98c46"}
Oct 30 17:20:57.013504 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:20:57.013467    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:20:57.031804 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:57.031775    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7"
Oct 30 17:20:57.047475 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:57.047447    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 30 17:20:57.047594 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:57.047507    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q" podUID="6906457b-d282-4603-8e72-edd173a4afa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 30 17:20:57.104870 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:57.102835    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh"
Oct 30 17:20:57.125930 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:57.125896    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh"
Oct 30 17:20:57.205423 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:57.205380    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-home\" (UniqueName: \"kubernetes.io/empty-dir/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-home\") pod \"4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe\" (UID: \"4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe\") "
Oct 30 17:20:57.205720 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:57.205698    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-home" pod "4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe" (UID: "4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:20:57.205777 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:57.205745    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-kube-api-access-77nl4\" (UniqueName: \"kubernetes.io/projected/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-kube-api-access-77nl4\") pod \"4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe\" (UID: \"4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe\") "
Oct 30 17:20:57.205814 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:57.205802    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-model-cache\") pod \"4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe\" (UID: \"4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe\") "
Oct 30 17:20:57.205855 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:57.205830    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-tmp-dir\") pod \"4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe\" (UID: \"4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe\") "
Oct 30 17:20:57.205920 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:57.205904    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-dshm\") pod \"4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe\" (UID: \"4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe\") "
Oct 30 17:20:57.205959 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:57.205936    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-tls-certs\") pod \"4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe\" (UID: \"4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe\") "
Oct 30 17:20:57.206096 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:57.206069    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-model-cache" pod "4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe" (UID: "4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:20:57.206236 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:57.206204    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-tmp-dir" pod "4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe" (UID: "4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:20:57.206738 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:57.206712    4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-model-cache\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:20:57.206860 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:57.206742    4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-tmp-dir\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:20:57.206860 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:57.206757    4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-home\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:20:57.208238 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:57.208213    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-kube-api-access-77nl4" pod "4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe" (UID: "4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe"). InnerVolumeSpecName "kube-api-access-77nl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:20:57.208674 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:57.208626    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-tls-certs" pod "4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe" (UID: "4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:20:57.208674 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:57.208627    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-dshm" pod "4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe" (UID: "4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:20:57.308497 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:57.308468    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77nl4\" (UniqueName: \"kubernetes.io/projected/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-kube-api-access-77nl4\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:20:57.308497 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:57.308496    4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-dshm\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:20:57.308667 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:57.308508    4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe-tls-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:20:57.796275 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:57.796228    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc"]
Oct 30 17:20:58.052528 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:58.050973    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7" event={"ID":"4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe","Type":"ContainerDied","Data":"62b32f981aca20cc1d896a7245270c000b57820fdc5fc3f8da751a0b2fd5f50a"}
Oct 30 17:20:58.085533 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:58.085497    4433 scope.go:122] "RemoveContainer" containerID="0ceb38e3f33ceff1771d1c8e56d54334b13322f739818972d787e197c5d98c46"
Oct 30 17:20:58.085923 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:58.057726    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7"
Oct 30 17:20:58.128710 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:58.058006    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 30 17:20:58.128710 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:58.123749    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q" podUID="6906457b-d282-4603-8e72-edd173a4afa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 30 17:20:58.229494 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:58.228427    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh"
Oct 30 17:20:58.270965 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:58.270931    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7"]
Oct 30 17:20:58.277131 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:58.277094    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d003a5/test-model-d003a5-kserve-5dddbf7989-pd4b7"]
Oct 30 17:20:59.048042 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:59.048004    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 30 17:20:59.048237 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:59.048067    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q" podUID="6906457b-d282-4603-8e72-edd173a4afa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 30 17:20:59.709863 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:20:59.709811    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe" path="/var/lib/kubelet/pods/4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe/volumes"
Oct 30 17:21:00.047585 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:00.047496    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 30 17:21:00.047585 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:00.047564    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q" podUID="6906457b-d282-4603-8e72-edd173a4afa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 30 17:21:01.052243 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:01.052211    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:21:01.052727 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:01.052706    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:21:01.053598 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:01.053576    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:21:11.466139 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:11.466096    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"]
Oct 30 17:21:11.466710 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:11.466358    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx" podUID="07938ade-58fb-4ca9-a7a2-618a0e318196" containerName="istio-proxy" containerID="cri-o://1c5137a1215d1e467e13febb7f6a64b231936a9d442dfd8270240708acc723a8" gracePeriod=30
Oct 30 17:21:11.679660 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:11.679626    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55"]
Oct 30 17:21:12.844609 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:12.844562    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc" podUID="36e810bc-53a4-46a7-9ccc-805373681902" containerName="main" containerID="cri-o://f838bbdb57b2b280ef873a79ab812ef5d2d1292b57666b9ab32931920b601499" gracePeriod=45
Oct 30 17:21:13.123790 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:13.123763    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc"
Oct 30 17:21:13.181969 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:13.181939    4433 generic.go:356] "Generic (PLEG): container finished" podID="36e810bc-53a4-46a7-9ccc-805373681902" containerID="f838bbdb57b2b280ef873a79ab812ef5d2d1292b57666b9ab32931920b601499" exitCode=0
Oct 30 17:21:13.182166 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:13.182011    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc"
Oct 30 17:21:13.182166 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:13.182034    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc" event={"ID":"36e810bc-53a4-46a7-9ccc-805373681902","Type":"ContainerDied","Data":"f838bbdb57b2b280ef873a79ab812ef5d2d1292b57666b9ab32931920b601499"}
Oct 30 17:21:13.182166 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:13.182085    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc" event={"ID":"36e810bc-53a4-46a7-9ccc-805373681902","Type":"ContainerDied","Data":"aedeacd74925d0a78c7a25c5f632b57fce170a9bea9a712e3da8588ccb8ec4b4"}
Oct 30 17:21:13.182166 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:13.182109    4433 scope.go:122] "RemoveContainer" containerID="f838bbdb57b2b280ef873a79ab812ef5d2d1292b57666b9ab32931920b601499"
Oct 30 17:21:13.192535 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:13.192508    4433 scope.go:122] "RemoveContainer" containerID="f838bbdb57b2b280ef873a79ab812ef5d2d1292b57666b9ab32931920b601499"
Oct 30 17:21:13.192823 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:21:13.192792    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f838bbdb57b2b280ef873a79ab812ef5d2d1292b57666b9ab32931920b601499\": container with ID starting with f838bbdb57b2b280ef873a79ab812ef5d2d1292b57666b9ab32931920b601499 not found: ID does not exist" containerID="f838bbdb57b2b280ef873a79ab812ef5d2d1292b57666b9ab32931920b601499"
Oct 30 17:21:13.192906 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:13.192831    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f838bbdb57b2b280ef873a79ab812ef5d2d1292b57666b9ab32931920b601499"} err="failed to get container status \"f838bbdb57b2b280ef873a79ab812ef5d2d1292b57666b9ab32931920b601499\": rpc error: code = NotFound desc = could not find container \"f838bbdb57b2b280ef873a79ab812ef5d2d1292b57666b9ab32931920b601499\": container with ID starting with f838bbdb57b2b280ef873a79ab812ef5d2d1292b57666b9ab32931920b601499 not found: ID does not exist"
Oct 30 17:21:13.244027 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:13.243999    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36e810bc-53a4-46a7-9ccc-805373681902-tls-certs\" (UniqueName: \"kubernetes.io/secret/36e810bc-53a4-46a7-9ccc-805373681902-tls-certs\") pod \"36e810bc-53a4-46a7-9ccc-805373681902\" (UID: \"36e810bc-53a4-46a7-9ccc-805373681902\") "
Oct 30 17:21:13.244187 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:13.244073    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36e810bc-53a4-46a7-9ccc-805373681902-kube-api-access-cmnzx\" (UniqueName: \"kubernetes.io/projected/36e810bc-53a4-46a7-9ccc-805373681902-kube-api-access-cmnzx\") pod \"36e810bc-53a4-46a7-9ccc-805373681902\" (UID: \"36e810bc-53a4-46a7-9ccc-805373681902\") "
Oct 30 17:21:13.244187 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:13.244137    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36e810bc-53a4-46a7-9ccc-805373681902-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36e810bc-53a4-46a7-9ccc-805373681902-model-cache\") pod \"36e810bc-53a4-46a7-9ccc-805373681902\" (UID: \"36e810bc-53a4-46a7-9ccc-805373681902\") "
Oct 30 17:21:13.244317 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:13.244213    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36e810bc-53a4-46a7-9ccc-805373681902-dshm\" (UniqueName: \"kubernetes.io/empty-dir/36e810bc-53a4-46a7-9ccc-805373681902-dshm\") pod \"36e810bc-53a4-46a7-9ccc-805373681902\" (UID: \"36e810bc-53a4-46a7-9ccc-805373681902\") "
Oct 30 17:21:13.244317 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:13.244295    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36e810bc-53a4-46a7-9ccc-805373681902-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36e810bc-53a4-46a7-9ccc-805373681902-tmp-dir\") pod \"36e810bc-53a4-46a7-9ccc-805373681902\" (UID: \"36e810bc-53a4-46a7-9ccc-805373681902\") "
Oct 30 17:21:13.244478 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:13.244440    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e810bc-53a4-46a7-9ccc-805373681902-model-cache" pod "36e810bc-53a4-46a7-9ccc-805373681902" (UID: "36e810bc-53a4-46a7-9ccc-805373681902"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:13.244478 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:13.244453    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36e810bc-53a4-46a7-9ccc-805373681902-home\" (UniqueName: \"kubernetes.io/empty-dir/36e810bc-53a4-46a7-9ccc-805373681902-home\") pod \"36e810bc-53a4-46a7-9ccc-805373681902\" (UID: \"36e810bc-53a4-46a7-9ccc-805373681902\") "
Oct 30 17:21:13.244680 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:13.244656    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e810bc-53a4-46a7-9ccc-805373681902-tmp-dir" pod "36e810bc-53a4-46a7-9ccc-805373681902" (UID: "36e810bc-53a4-46a7-9ccc-805373681902"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:13.244751 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:13.244733    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e810bc-53a4-46a7-9ccc-805373681902-home" pod "36e810bc-53a4-46a7-9ccc-805373681902" (UID: "36e810bc-53a4-46a7-9ccc-805373681902"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:13.245251 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:13.245226    4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36e810bc-53a4-46a7-9ccc-805373681902-model-cache\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:21:13.245335 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:13.245256    4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36e810bc-53a4-46a7-9ccc-805373681902-tmp-dir\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:21:13.245335 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:13.245272    4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36e810bc-53a4-46a7-9ccc-805373681902-home\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:21:13.246364 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:13.246333    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e810bc-53a4-46a7-9ccc-805373681902-dshm" pod "36e810bc-53a4-46a7-9ccc-805373681902" (UID: "36e810bc-53a4-46a7-9ccc-805373681902"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:13.246364 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:13.246354    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36e810bc-53a4-46a7-9ccc-805373681902-tls-certs" pod "36e810bc-53a4-46a7-9ccc-805373681902" (UID: "36e810bc-53a4-46a7-9ccc-805373681902"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:21:13.246536 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:13.246465    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36e810bc-53a4-46a7-9ccc-805373681902-kube-api-access-cmnzx" pod "36e810bc-53a4-46a7-9ccc-805373681902" (UID: "36e810bc-53a4-46a7-9ccc-805373681902"). InnerVolumeSpecName "kube-api-access-cmnzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:21:13.346724 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:13.346685    4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36e810bc-53a4-46a7-9ccc-805373681902-dshm\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:21:13.346724 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:13.346712    4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36e810bc-53a4-46a7-9ccc-805373681902-tls-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:21:13.346724 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:13.346726    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmnzx\" (UniqueName: \"kubernetes.io/projected/36e810bc-53a4-46a7-9ccc-805373681902-kube-api-access-cmnzx\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:21:13.529046 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:13.529009    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc"]
Oct 30 17:21:13.532506 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:13.532478    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-5d67ca/test-model-5d67ca-kserve-7595bf8795-m77qc"]
Oct 30 17:21:13.697482 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:13.697443    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36e810bc-53a4-46a7-9ccc-805373681902" path="/var/lib/kubelet/pods/36e810bc-53a4-46a7-9ccc-805373681902/volumes"
Oct 30 17:21:16.353081 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:16.353037    4433 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:21:16.353508 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:16.353114    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx" podUID="07938ade-58fb-4ca9-a7a2-618a0e318196" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:21:16.761845 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:16.761811    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"
Oct 30 17:21:16.802919 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:16.802886    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07938ade-58fb-4ca9-a7a2-618a0e318196-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07938ade-58fb-4ca9-a7a2-618a0e318196-istio-podinfo\") pod \"07938ade-58fb-4ca9-a7a2-618a0e318196\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") "
Oct 30 17:21:16.803112 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:16.802941    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07938ade-58fb-4ca9-a7a2-618a0e318196-istio-token\" (UniqueName: \"kubernetes.io/projected/07938ade-58fb-4ca9-a7a2-618a0e318196-istio-token\") pod \"07938ade-58fb-4ca9-a7a2-618a0e318196\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") "
Oct 30 17:21:16.803112 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:16.802994    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07938ade-58fb-4ca9-a7a2-618a0e318196-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07938ade-58fb-4ca9-a7a2-618a0e318196-istio-envoy\") pod \"07938ade-58fb-4ca9-a7a2-618a0e318196\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") "
Oct 30 17:21:16.803112 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:16.803018    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07938ade-58fb-4ca9-a7a2-618a0e318196-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07938ade-58fb-4ca9-a7a2-618a0e318196-istio-data\") pod \"07938ade-58fb-4ca9-a7a2-618a0e318196\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") "
Oct 30 17:21:16.803112 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:16.803097    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07938ade-58fb-4ca9-a7a2-618a0e318196-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07938ade-58fb-4ca9-a7a2-618a0e318196-workload-certs\") pod \"07938ade-58fb-4ca9-a7a2-618a0e318196\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") "
Oct 30 17:21:16.803333 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:16.803173    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07938ade-58fb-4ca9-a7a2-618a0e318196-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07938ade-58fb-4ca9-a7a2-618a0e318196-istiod-ca-cert\") pod \"07938ade-58fb-4ca9-a7a2-618a0e318196\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") "
Oct 30 17:21:16.803333 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:16.803307    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07938ade-58fb-4ca9-a7a2-618a0e318196-kube-api-access-q26g9\" (UniqueName: \"kubernetes.io/projected/07938ade-58fb-4ca9-a7a2-618a0e318196-kube-api-access-q26g9\") pod \"07938ade-58fb-4ca9-a7a2-618a0e318196\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") "
Oct 30 17:21:16.803470 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:16.803386    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07938ade-58fb-4ca9-a7a2-618a0e318196-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07938ade-58fb-4ca9-a7a2-618a0e318196-workload-socket\") pod \"07938ade-58fb-4ca9-a7a2-618a0e318196\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") "
Oct 30 17:21:16.803470 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:16.803369    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07938ade-58fb-4ca9-a7a2-618a0e318196-istio-data" pod "07938ade-58fb-4ca9-a7a2-618a0e318196" (UID: "07938ade-58fb-4ca9-a7a2-618a0e318196"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:16.803574 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:16.803462    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07938ade-58fb-4ca9-a7a2-618a0e318196-workload-certs" pod "07938ade-58fb-4ca9-a7a2-618a0e318196" (UID: "07938ade-58fb-4ca9-a7a2-618a0e318196"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:16.803629 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:16.803609    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07938ade-58fb-4ca9-a7a2-618a0e318196-istiod-ca-cert" pod "07938ade-58fb-4ca9-a7a2-618a0e318196" (UID: "07938ade-58fb-4ca9-a7a2-618a0e318196"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:21:16.803682 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:16.803617    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07938ade-58fb-4ca9-a7a2-618a0e318196-workload-socket" pod "07938ade-58fb-4ca9-a7a2-618a0e318196" (UID: "07938ade-58fb-4ca9-a7a2-618a0e318196"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:16.804271 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:16.804245    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07938ade-58fb-4ca9-a7a2-618a0e318196-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:21:16.804385 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:16.804273    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07938ade-58fb-4ca9-a7a2-618a0e318196-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:21:16.804385 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:16.804287    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07938ade-58fb-4ca9-a7a2-618a0e318196-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:21:16.804385 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:16.804302    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07938ade-58fb-4ca9-a7a2-618a0e318196-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:21:16.805463 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:16.805397    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07938ade-58fb-4ca9-a7a2-618a0e318196-istio-podinfo" pod "07938ade-58fb-4ca9-a7a2-618a0e318196" (UID: "07938ade-58fb-4ca9-a7a2-618a0e318196"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:21:16.805463 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:16.805392    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07938ade-58fb-4ca9-a7a2-618a0e318196-istio-envoy" pod "07938ade-58fb-4ca9-a7a2-618a0e318196" (UID: "07938ade-58fb-4ca9-a7a2-618a0e318196"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:16.805618 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:16.805480    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07938ade-58fb-4ca9-a7a2-618a0e318196-istio-token" pod "07938ade-58fb-4ca9-a7a2-618a0e318196" (UID: "07938ade-58fb-4ca9-a7a2-618a0e318196"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:21:16.805618 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:16.805548    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07938ade-58fb-4ca9-a7a2-618a0e318196-kube-api-access-q26g9" pod "07938ade-58fb-4ca9-a7a2-618a0e318196" (UID: "07938ade-58fb-4ca9-a7a2-618a0e318196"). InnerVolumeSpecName "kube-api-access-q26g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:21:16.905140 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:16.905050    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07938ade-58fb-4ca9-a7a2-618a0e318196-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07938ade-58fb-4ca9-a7a2-618a0e318196-credential-socket\") pod \"07938ade-58fb-4ca9-a7a2-618a0e318196\" (UID: \"07938ade-58fb-4ca9-a7a2-618a0e318196\") "
Oct 30 17:21:16.905379 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:16.905353    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07938ade-58fb-4ca9-a7a2-618a0e318196-credential-socket" pod "07938ade-58fb-4ca9-a7a2-618a0e318196" (UID: "07938ade-58fb-4ca9-a7a2-618a0e318196"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:16.905755 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:16.905738    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07938ade-58fb-4ca9-a7a2-618a0e318196-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:21:16.905791 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:16.905762    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q26g9\" (UniqueName: \"kubernetes.io/projected/07938ade-58fb-4ca9-a7a2-618a0e318196-kube-api-access-q26g9\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:21:16.905791 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:16.905779    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07938ade-58fb-4ca9-a7a2-618a0e318196-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:21:16.905855 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:16.905792    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07938ade-58fb-4ca9-a7a2-618a0e318196-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:21:16.905855 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:16.905806    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07938ade-58fb-4ca9-a7a2-618a0e318196-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:21:17.212366 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:17.212335    4433 generic.go:356] "Generic (PLEG): container finished" podID="07938ade-58fb-4ca9-a7a2-618a0e318196" containerID="1c5137a1215d1e467e13febb7f6a64b231936a9d442dfd8270240708acc723a8" exitCode=0
Oct 30 17:21:17.212607 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:17.212378    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"
Oct 30 17:21:17.212607 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:17.212447    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx" event={"ID":"07938ade-58fb-4ca9-a7a2-618a0e318196","Type":"ContainerDied","Data":"1c5137a1215d1e467e13febb7f6a64b231936a9d442dfd8270240708acc723a8"}
Oct 30 17:21:17.212607 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:17.212503    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx" event={"ID":"07938ade-58fb-4ca9-a7a2-618a0e318196","Type":"ContainerDied","Data":"d21abd95eb648d23e431633accec7d34d377fc2e3e8876f853e35cd02cf4f9f0"}
Oct 30 17:21:17.212607 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:17.212528    4433 scope.go:122] "RemoveContainer" containerID="1c5137a1215d1e467e13febb7f6a64b231936a9d442dfd8270240708acc723a8"
Oct 30 17:21:17.223666 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:17.223604    4433 scope.go:122] "RemoveContainer" containerID="1c5137a1215d1e467e13febb7f6a64b231936a9d442dfd8270240708acc723a8"
Oct 30 17:21:17.223968 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:21:17.223949    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c5137a1215d1e467e13febb7f6a64b231936a9d442dfd8270240708acc723a8\": container with ID starting with 1c5137a1215d1e467e13febb7f6a64b231936a9d442dfd8270240708acc723a8 not found: ID does not exist" containerID="1c5137a1215d1e467e13febb7f6a64b231936a9d442dfd8270240708acc723a8"
Oct 30 17:21:17.224057 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:17.223979    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c5137a1215d1e467e13febb7f6a64b231936a9d442dfd8270240708acc723a8"} err="failed to get container status \"1c5137a1215d1e467e13febb7f6a64b231936a9d442dfd8270240708acc723a8\": rpc error: code = NotFound desc = could not find container \"1c5137a1215d1e467e13febb7f6a64b231936a9d442dfd8270240708acc723a8\": container with ID starting with 1c5137a1215d1e467e13febb7f6a64b231936a9d442dfd8270240708acc723a8 not found: ID does not exist"
Oct 30 17:21:17.264721 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:17.264687    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"]
Oct 30 17:21:17.270224 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:17.270198    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jtkcx"]
Oct 30 17:21:17.696606 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:17.696567    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07938ade-58fb-4ca9-a7a2-618a0e318196" path="/var/lib/kubelet/pods/07938ade-58fb-4ca9-a7a2-618a0e318196/volumes"
Oct 30 17:21:20.533669 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.533628    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"]
Oct 30 17:21:20.534499 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.534482    4433 cpu_manager.go:414] "containerMap: removing container" podUID="07938ade-58fb-4ca9-a7a2-618a0e318196" containerName="istio-proxy"
Oct 30 17:21:20.534577 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.534500    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07938ade-58fb-4ca9-a7a2-618a0e318196" containerName="istio-proxy"
Oct 30 17:21:20.534577 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.534557    4433 cpu_manager.go:414] "containerMap: removing container" podUID="36e810bc-53a4-46a7-9ccc-805373681902" containerName="main"
Oct 30 17:21:20.534577 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.534565    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36e810bc-53a4-46a7-9ccc-805373681902" containerName="main"
Oct 30 17:21:20.534684 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.534582    4433 cpu_manager.go:414] "containerMap: removing container" podUID="4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe" containerName="main"
Oct 30 17:21:20.534684 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.534589    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe" containerName="main"
Oct 30 17:21:20.534775 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.534763    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="36e810bc-53a4-46a7-9ccc-805373681902" containerName="main"
Oct 30 17:21:20.534809 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.534780    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a3789e1-4a1d-40af-ab29-1c7e7b2f4abe" containerName="main"
Oct 30 17:21:20.534809 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.534795    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="07938ade-58fb-4ca9-a7a2-618a0e318196" containerName="istio-proxy"
Oct 30 17:21:20.540462 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.540437    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"
Oct 30 17:21:20.551241 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.551217    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"]
Oct 30 17:21:20.555308 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.555240    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"
Oct 30 17:21:20.563437 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.559199    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d003a5-openshift-default-dockercfg-8dqnq\""
Oct 30 17:21:20.564583 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.564562    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"]
Oct 30 17:21:20.595803 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.590509    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"]
Oct 30 17:21:20.660015 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.659983    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6w54\" (UniqueName: \"kubernetes.io/projected/bb49dfad-1e85-4b96-9dd8-42374a708643-kube-api-access-f6w54\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"
Oct 30 17:21:20.660206 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.660038    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb49dfad-1e85-4b96-9dd8-42374a708643-istio-podinfo\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"
Oct 30 17:21:20.660206 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.660104    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jrqm\" (UniqueName: \"kubernetes.io/projected/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-kube-api-access-4jrqm\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"
Oct 30 17:21:20.660206 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.660131    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb49dfad-1e85-4b96-9dd8-42374a708643-istio-token\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"
Oct 30 17:21:20.660350 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.660251    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-workload-socket\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"
Oct 30 17:21:20.660350 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.660284    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb49dfad-1e85-4b96-9dd8-42374a708643-workload-certs\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"
Oct 30 17:21:20.660350 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.660308    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-istio-data\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"
Oct 30 17:21:20.660478 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.660363    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb49dfad-1e85-4b96-9dd8-42374a708643-workload-socket\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"
Oct 30 17:21:20.660478 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.660422    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-istio-token\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"
Oct 30 17:21:20.660478 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.660450    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb49dfad-1e85-4b96-9dd8-42374a708643-istio-envoy\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"
Oct 30 17:21:20.660620 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.660500    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb49dfad-1e85-4b96-9dd8-42374a708643-credential-socket\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"
Oct 30 17:21:20.660620 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.660586    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb49dfad-1e85-4b96-9dd8-42374a708643-istio-data\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"
Oct 30 17:21:20.660729 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.660681    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-istio-podinfo\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"
Oct 30 17:21:20.660784 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.660729    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb49dfad-1e85-4b96-9dd8-42374a708643-istiod-ca-cert\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"
Oct 30 17:21:20.660825 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.660797    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-credential-socket\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"
Oct 30 17:21:20.660867 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.660823    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-workload-certs\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"
Oct 30 17:21:20.660909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.660890    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-istio-envoy\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"
Oct 30 17:21:20.660947 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.660914    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-istiod-ca-cert\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"
Oct 30 17:21:20.761862 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.761829    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-istio-data\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"
Oct 30 17:21:20.762044 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.761874    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb49dfad-1e85-4b96-9dd8-42374a708643-workload-socket\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"
Oct 30 17:21:20.762044 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.761914    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-istio-token\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"
Oct 30 17:21:20.762044 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.761943    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb49dfad-1e85-4b96-9dd8-42374a708643-istio-envoy\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"
Oct 30 17:21:20.762213 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.762044    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb49dfad-1e85-4b96-9dd8-42374a708643-credential-socket\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"
Oct 30 17:21:20.762213 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.762075    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb49dfad-1e85-4b96-9dd8-42374a708643-istio-data\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"
Oct 30 17:21:20.762213 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.762150    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-istio-podinfo\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"
Oct 30 17:21:20.762213 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.762205    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb49dfad-1e85-4b96-9dd8-42374a708643-istiod-ca-cert\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"
Oct 30 17:21:20.762440 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.762242    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-istio-data\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"
Oct 30 17:21:20.762440 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.762278    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-credential-socket\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"
Oct 30 17:21:20.762440 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.762307    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-workload-certs\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"
Oct 30 17:21:20.762440 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.762355    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb49dfad-1e85-4b96-9dd8-42374a708643-workload-socket\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"
Oct 30 17:21:20.762440 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.762387    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-istio-envoy\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"
Oct 30 17:21:20.762715 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.762468    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-istiod-ca-cert\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"
Oct 30 17:21:20.762715 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.762514    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6w54\" (UniqueName: \"kubernetes.io/projected/bb49dfad-1e85-4b96-9dd8-42374a708643-kube-api-access-f6w54\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"
Oct 30 17:21:20.762715 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.762589    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-workload-certs\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"
Oct 30 17:21:20.762715 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.762593    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb49dfad-1e85-4b96-9dd8-42374a708643-istio-podinfo\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"
Oct 30 17:21:20.762715 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.762636    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jrqm\" (UniqueName: \"kubernetes.io/projected/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-kube-api-access-4jrqm\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"
Oct 30 17:21:20.762715 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.762686    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb49dfad-1e85-4b96-9dd8-42374a708643-istio-token\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"
Oct 30 17:21:20.763036 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.762828    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-workload-socket\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"
Oct 30 17:21:20.763036 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.762886    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb49dfad-1e85-4b96-9dd8-42374a708643-workload-certs\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"
Oct 30 17:21:20.763036 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.762918    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb49dfad-1e85-4b96-9dd8-42374a708643-istio-data\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"
Oct 30 17:21:20.763036 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.762638    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb49dfad-1e85-4b96-9dd8-42374a708643-credential-socket\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"
Oct 30 17:21:20.763243 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.763153    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb49dfad-1e85-4b96-9dd8-42374a708643-istiod-ca-cert\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"
Oct 30 17:21:20.763243 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.763212    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb49dfad-1e85-4b96-9dd8-42374a708643-workload-certs\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"
Oct 30 17:21:20.763441 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.763416    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-credential-socket\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"
Oct 30 17:21:20.763629 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.763606    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-workload-socket\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"
Oct 30 17:21:20.763864 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.763833    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-istiod-ca-cert\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"
Oct 30 17:21:20.764878 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.764846    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb49dfad-1e85-4b96-9dd8-42374a708643-istio-envoy\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"
Oct 30 17:21:20.765167 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.765141    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb49dfad-1e85-4b96-9dd8-42374a708643-istio-podinfo\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"
Oct 30 17:21:20.765388 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.765372    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-istio-podinfo\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"
Oct 30 17:21:20.765479 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.765379    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-istio-envoy\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"
Oct 30 17:21:20.785520 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.785455    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb49dfad-1e85-4b96-9dd8-42374a708643-istio-token\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"
Oct 30 17:21:20.785520 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.785493    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-istio-token\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"
Oct 30 17:21:20.785672 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.785592    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jrqm\" (UniqueName: \"kubernetes.io/projected/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-kube-api-access-4jrqm\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"
Oct 30 17:21:20.785971 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.785952    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6w54\" (UniqueName: \"kubernetes.io/projected/bb49dfad-1e85-4b96-9dd8-42374a708643-kube-api-access-f6w54\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"
Oct 30 17:21:20.855662 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.855630    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"
Oct 30 17:21:20.890506 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:20.890475    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"
Oct 30 17:21:21.177878 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:21.177847    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"]
Oct 30 17:21:21.179166 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:21:21.179141    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc07529e2_9bcd_4ec9_9c98_e0bb48c8effb.slice/crio-e3d79ec5ee458c5aec08d6d962f2db21858aef475734cf11f09e5db2939002dd WatchSource:0}: Error finding container e3d79ec5ee458c5aec08d6d962f2db21858aef475734cf11f09e5db2939002dd: Status 404 returned error can't find the container with id e3d79ec5ee458c5aec08d6d962f2db21858aef475734cf11f09e5db2939002dd
Oct 30 17:21:21.182242 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:21.182209    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:21:21.182320 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:21.182278    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:21:21.182320 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:21.182308    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:21:21.221050 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:21.221025    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"]
Oct 30 17:21:21.222860 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:21:21.222829    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb49dfad_1e85_4b96_9dd8_42374a708643.slice/crio-b5905610ee7ed00d35449863f8a56e44d469d5d01d98a52b5be5368c4dcf6967 WatchSource:0}: Error finding container b5905610ee7ed00d35449863f8a56e44d469d5d01d98a52b5be5368c4dcf6967: Status 404 returned error can't find the container with id b5905610ee7ed00d35449863f8a56e44d469d5d01d98a52b5be5368c4dcf6967
Oct 30 17:21:21.226054 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:21.226023    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:21:21.226141 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:21.226088    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:21:21.226141 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:21.226116    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:21:21.242004 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:21.241975    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5" event={"ID":"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb","Type":"ContainerStarted","Data":"e3d79ec5ee458c5aec08d6d962f2db21858aef475734cf11f09e5db2939002dd"}
Oct 30 17:21:21.243446 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:21.243427    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx" event={"ID":"bb49dfad-1e85-4b96-9dd8-42374a708643","Type":"ContainerStarted","Data":"b5905610ee7ed00d35449863f8a56e44d469d5d01d98a52b5be5368c4dcf6967"}
Oct 30 17:21:22.358669 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:22.346760    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5" event={"ID":"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb","Type":"ContainerStarted","Data":"77dcb683431ee8eaaaa39784b6d93f534818ac8768b96608a977d28390f7032f"}
Oct 30 17:21:22.381429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:22.360747    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx" event={"ID":"bb49dfad-1e85-4b96-9dd8-42374a708643","Type":"ContainerStarted","Data":"32eba274730c1cae86773f4ec9032df5676779149c5b8b78ece6e0af3deff139"}
Oct 30 17:21:22.528069 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:22.528002    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5" podStartSLOduration=2.527982695 podStartE2EDuration="2.527982695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:21:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:21:22.496267682 +0000 UTC m=+2058.931169888" watchObservedRunningTime="2026-10-30 17:21:22.527982695 +0000 UTC m=+2058.962884904"
Oct 30 17:21:22.871428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:22.859750    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"
Oct 30 17:21:22.871428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:22.859905    4433 patch_prober.go:29] interesting pod/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 17:21:22.871428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:22.859962    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5" podUID="c07529e2-9bcd-4ec9-9c98-e0bb48c8effb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 17:21:22.896427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:22.893115    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"
Oct 30 17:21:22.896427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:22.893254    4433 patch_prober.go:29] interesting pod/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 17:21:22.896427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:22.893297    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx" podUID="bb49dfad-1e85-4b96-9dd8-42374a708643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 17:21:23.866426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:23.865374    4433 patch_prober.go:29] interesting pod/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 17:21:23.866426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:23.865456    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5" podUID="c07529e2-9bcd-4ec9-9c98-e0bb48c8effb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 17:21:23.893439 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:23.891604    4433 patch_prober.go:29] interesting pod/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 17:21:23.893439 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:23.891664    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx" podUID="bb49dfad-1e85-4b96-9dd8-42374a708643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 17:21:24.866430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:24.863236    4433 patch_prober.go:29] interesting pod/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 17:21:24.866430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:24.863330    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5" podUID="c07529e2-9bcd-4ec9-9c98-e0bb48c8effb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 17:21:24.893884 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:24.893840    4433 patch_prober.go:29] interesting pod/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 17:21:24.894214 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:24.894180    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx" podUID="bb49dfad-1e85-4b96-9dd8-42374a708643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 17:21:25.864427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:25.863540    4433 patch_prober.go:29] interesting pod/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 17:21:25.864427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:25.863611    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5" podUID="c07529e2-9bcd-4ec9-9c98-e0bb48c8effb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 17:21:25.893426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:25.891088    4433 patch_prober.go:29] interesting pod/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 17:21:25.893426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:25.891146    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx" podUID="bb49dfad-1e85-4b96-9dd8-42374a708643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 17:21:26.706713 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:26.706642    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55" podUID="8f245c01-8039-4ce3-a4de-ce744f348adc" containerName="main" containerID="cri-o://5420b797c0a27f6734aea206a4d484be716351c99bf947d07b7383fc33e60aa3" gracePeriod=45
Oct 30 17:21:26.872431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:26.859058    4433 patch_prober.go:29] interesting pod/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 17:21:26.872431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:26.859121    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5" podUID="c07529e2-9bcd-4ec9-9c98-e0bb48c8effb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 17:21:26.903431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:26.893508    4433 patch_prober.go:29] interesting pod/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 17:21:26.903431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:26.893574    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx" podUID="bb49dfad-1e85-4b96-9dd8-42374a708643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 17:21:27.114708 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:21:27.114609    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:21:27.200482 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:27.197622    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55"
Oct 30 17:21:27.230867 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:27.230837    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f245c01-8039-4ce3-a4de-ce744f348adc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f245c01-8039-4ce3-a4de-ce744f348adc-tmp-dir\") pod \"8f245c01-8039-4ce3-a4de-ce744f348adc\" (UID: \"8f245c01-8039-4ce3-a4de-ce744f348adc\") "
Oct 30 17:21:27.231067 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:27.230945    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f245c01-8039-4ce3-a4de-ce744f348adc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f245c01-8039-4ce3-a4de-ce744f348adc-dshm\") pod \"8f245c01-8039-4ce3-a4de-ce744f348adc\" (UID: \"8f245c01-8039-4ce3-a4de-ce744f348adc\") "
Oct 30 17:21:27.231067 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:27.231044    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f245c01-8039-4ce3-a4de-ce744f348adc-home\" (UniqueName: \"kubernetes.io/empty-dir/8f245c01-8039-4ce3-a4de-ce744f348adc-home\") pod \"8f245c01-8039-4ce3-a4de-ce744f348adc\" (UID: \"8f245c01-8039-4ce3-a4de-ce744f348adc\") "
Oct 30 17:21:27.231207 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:27.231092    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f245c01-8039-4ce3-a4de-ce744f348adc-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f245c01-8039-4ce3-a4de-ce744f348adc-tls-certs\") pod \"8f245c01-8039-4ce3-a4de-ce744f348adc\" (UID: \"8f245c01-8039-4ce3-a4de-ce744f348adc\") "
Oct 30 17:21:27.231207 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:27.231134    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f245c01-8039-4ce3-a4de-ce744f348adc-tmp-dir" pod "8f245c01-8039-4ce3-a4de-ce744f348adc" (UID: "8f245c01-8039-4ce3-a4de-ce744f348adc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:27.231207 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:27.231132    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f245c01-8039-4ce3-a4de-ce744f348adc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f245c01-8039-4ce3-a4de-ce744f348adc-model-cache\") pod \"8f245c01-8039-4ce3-a4de-ce744f348adc\" (UID: \"8f245c01-8039-4ce3-a4de-ce744f348adc\") "
Oct 30 17:21:27.231356 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:27.231263    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f245c01-8039-4ce3-a4de-ce744f348adc-kube-api-access-tzps4\" (UniqueName: \"kubernetes.io/projected/8f245c01-8039-4ce3-a4de-ce744f348adc-kube-api-access-tzps4\") pod \"8f245c01-8039-4ce3-a4de-ce744f348adc\" (UID: \"8f245c01-8039-4ce3-a4de-ce744f348adc\") "
Oct 30 17:21:27.231356 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:27.231341    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f245c01-8039-4ce3-a4de-ce744f348adc-home" pod "8f245c01-8039-4ce3-a4de-ce744f348adc" (UID: "8f245c01-8039-4ce3-a4de-ce744f348adc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:27.231597 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:27.231570    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f245c01-8039-4ce3-a4de-ce744f348adc-model-cache" pod "8f245c01-8039-4ce3-a4de-ce744f348adc" (UID: "8f245c01-8039-4ce3-a4de-ce744f348adc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:27.232117 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:27.232095    4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f245c01-8039-4ce3-a4de-ce744f348adc-tmp-dir\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:21:27.232117 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:27.232117    4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f245c01-8039-4ce3-a4de-ce744f348adc-home\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:21:27.232296 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:27.232125    4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f245c01-8039-4ce3-a4de-ce744f348adc-model-cache\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:21:27.233762 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:27.233625    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f245c01-8039-4ce3-a4de-ce744f348adc-tls-certs" pod "8f245c01-8039-4ce3-a4de-ce744f348adc" (UID: "8f245c01-8039-4ce3-a4de-ce744f348adc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:21:27.233762 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:27.233717    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f245c01-8039-4ce3-a4de-ce744f348adc-kube-api-access-tzps4" pod "8f245c01-8039-4ce3-a4de-ce744f348adc" (UID: "8f245c01-8039-4ce3-a4de-ce744f348adc"). InnerVolumeSpecName "kube-api-access-tzps4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:21:27.237507 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:27.237485    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f245c01-8039-4ce3-a4de-ce744f348adc-dshm" pod "8f245c01-8039-4ce3-a4de-ce744f348adc" (UID: "8f245c01-8039-4ce3-a4de-ce744f348adc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:27.240095 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:27.240048    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx" podStartSLOduration=7.240032474 podStartE2EDuration="7.240032474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:21:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:21:22.570026639 +0000 UTC m=+2059.004928846" watchObservedRunningTime="2026-10-30 17:21:27.240032474 +0000 UTC m=+2063.674934683"
Oct 30 17:21:27.334494 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:27.334454    4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f245c01-8039-4ce3-a4de-ce744f348adc-tls-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:21:27.334494 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:27.334494    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzps4\" (UniqueName: \"kubernetes.io/projected/8f245c01-8039-4ce3-a4de-ce744f348adc-kube-api-access-tzps4\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:21:27.334755 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:27.334508    4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f245c01-8039-4ce3-a4de-ce744f348adc-dshm\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:21:27.528226 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:27.528188    4433 generic.go:356] "Generic (PLEG): container finished" podID="8f245c01-8039-4ce3-a4de-ce744f348adc" containerID="5420b797c0a27f6734aea206a4d484be716351c99bf947d07b7383fc33e60aa3" exitCode=0
Oct 30 17:21:27.528419 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:27.528255    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55"
Oct 30 17:21:27.528419 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:27.528277    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55" event={"ID":"8f245c01-8039-4ce3-a4de-ce744f348adc","Type":"ContainerDied","Data":"5420b797c0a27f6734aea206a4d484be716351c99bf947d07b7383fc33e60aa3"}
Oct 30 17:21:27.528419 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:27.528324    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55" event={"ID":"8f245c01-8039-4ce3-a4de-ce744f348adc","Type":"ContainerDied","Data":"35fa446cc05779abbe28aa19427c08a338d4df51a1ea96dc51407f2738248619"}
Oct 30 17:21:27.528419 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:27.528341    4433 scope.go:122] "RemoveContainer" containerID="5420b797c0a27f6734aea206a4d484be716351c99bf947d07b7383fc33e60aa3"
Oct 30 17:21:27.571696 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:27.560933    4433 scope.go:122] "RemoveContainer" containerID="5420b797c0a27f6734aea206a4d484be716351c99bf947d07b7383fc33e60aa3"
Oct 30 17:21:27.590450 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:21:27.573493    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5420b797c0a27f6734aea206a4d484be716351c99bf947d07b7383fc33e60aa3\": container with ID starting with 5420b797c0a27f6734aea206a4d484be716351c99bf947d07b7383fc33e60aa3 not found: ID does not exist" containerID="5420b797c0a27f6734aea206a4d484be716351c99bf947d07b7383fc33e60aa3"
Oct 30 17:21:27.590450 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:27.573549    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5420b797c0a27f6734aea206a4d484be716351c99bf947d07b7383fc33e60aa3"} err="failed to get container status \"5420b797c0a27f6734aea206a4d484be716351c99bf947d07b7383fc33e60aa3\": rpc error: code = NotFound desc = could not find container \"5420b797c0a27f6734aea206a4d484be716351c99bf947d07b7383fc33e60aa3\": container with ID starting with 5420b797c0a27f6734aea206a4d484be716351c99bf947d07b7383fc33e60aa3 not found: ID does not exist"
Oct 30 17:21:27.623507 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:27.622710    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55"]
Oct 30 17:21:27.628790 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:27.628756    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c82666/test-model-c82666-kserve-686458489c-fvn55"]
Oct 30 17:21:27.711422 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:27.711375    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f245c01-8039-4ce3-a4de-ce744f348adc" path="/var/lib/kubelet/pods/8f245c01-8039-4ce3-a4de-ce744f348adc/volumes"
Oct 30 17:21:27.878482 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:27.873901    4433 patch_prober.go:29] interesting pod/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 17:21:27.878482 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:27.873970    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5" podUID="c07529e2-9bcd-4ec9-9c98-e0bb48c8effb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 17:21:27.900431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:27.896350    4433 patch_prober.go:29] interesting pod/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 17:21:27.900431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:27.896450    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx" podUID="bb49dfad-1e85-4b96-9dd8-42374a708643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 17:21:29.407069 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:29.407029    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"
Oct 30 17:21:29.408118 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:29.408040    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"
Oct 30 17:21:29.410339 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:29.410317    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"
Oct 30 17:21:29.567449 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:29.566804    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"
Oct 30 17:21:29.569353 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:29.567703    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"
Oct 30 17:21:29.569353 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:29.568670    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"
Oct 30 17:21:34.086532 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:34.085913    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"]
Oct 30 17:21:34.086532 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:34.086132    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf" podUID="e50e437f-d457-49cc-94fc-73aa7340721d" containerName="istio-proxy" containerID="cri-o://cdd767862445376e4927e819feb3ca722099f6b27d26ad271b2f0c9ada783b68" gracePeriod=30
Oct 30 17:21:35.249233 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:35.249197    4433 patch_prober.go:29] interesting pod/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:21:35.249646 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:35.249254    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf" podUID="e50e437f-d457-49cc-94fc-73aa7340721d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:21:39.422739 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.422713    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"
Oct 30 17:21:39.499626 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.499587    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e50e437f-d457-49cc-94fc-73aa7340721d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50e437f-d457-49cc-94fc-73aa7340721d-workload-socket\") pod \"e50e437f-d457-49cc-94fc-73aa7340721d\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") "
Oct 30 17:21:39.499832 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.499770    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e50e437f-d457-49cc-94fc-73aa7340721d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e50e437f-d457-49cc-94fc-73aa7340721d-istio-data\") pod \"e50e437f-d457-49cc-94fc-73aa7340721d\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") "
Oct 30 17:21:39.499832 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.499819    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e50e437f-d457-49cc-94fc-73aa7340721d-kube-api-access-8d5mz\" (UniqueName: \"kubernetes.io/projected/e50e437f-d457-49cc-94fc-73aa7340721d-kube-api-access-8d5mz\") pod \"e50e437f-d457-49cc-94fc-73aa7340721d\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") "
Oct 30 17:21:39.499980 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.499885    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e50e437f-d457-49cc-94fc-73aa7340721d-istio-token\" (UniqueName: \"kubernetes.io/projected/e50e437f-d457-49cc-94fc-73aa7340721d-istio-token\") pod \"e50e437f-d457-49cc-94fc-73aa7340721d\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") "
Oct 30 17:21:39.499980 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.499941    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e50e437f-d457-49cc-94fc-73aa7340721d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e50e437f-d457-49cc-94fc-73aa7340721d-istiod-ca-cert\") pod \"e50e437f-d457-49cc-94fc-73aa7340721d\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") "
Oct 30 17:21:39.499980 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.499941    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e50e437f-d457-49cc-94fc-73aa7340721d-workload-socket" pod "e50e437f-d457-49cc-94fc-73aa7340721d" (UID: "e50e437f-d457-49cc-94fc-73aa7340721d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:39.500165 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.500025    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e50e437f-d457-49cc-94fc-73aa7340721d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e50e437f-d457-49cc-94fc-73aa7340721d-istio-podinfo\") pod \"e50e437f-d457-49cc-94fc-73aa7340721d\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") "
Oct 30 17:21:39.500165 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.500069    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e50e437f-d457-49cc-94fc-73aa7340721d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50e437f-d457-49cc-94fc-73aa7340721d-credential-socket\") pod \"e50e437f-d457-49cc-94fc-73aa7340721d\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") "
Oct 30 17:21:39.500165 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.500138    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e50e437f-d457-49cc-94fc-73aa7340721d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e50e437f-d457-49cc-94fc-73aa7340721d-istio-envoy\") pod \"e50e437f-d457-49cc-94fc-73aa7340721d\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") "
Oct 30 17:21:39.500323 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.500181    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e50e437f-d457-49cc-94fc-73aa7340721d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e50e437f-d457-49cc-94fc-73aa7340721d-workload-certs\") pod \"e50e437f-d457-49cc-94fc-73aa7340721d\" (UID: \"e50e437f-d457-49cc-94fc-73aa7340721d\") "
Oct 30 17:21:39.500821 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.500666    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e50e437f-d457-49cc-94fc-73aa7340721d-istio-data" pod "e50e437f-d457-49cc-94fc-73aa7340721d" (UID: "e50e437f-d457-49cc-94fc-73aa7340721d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:39.500999 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.500925    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e50e437f-d457-49cc-94fc-73aa7340721d-workload-certs" pod "e50e437f-d457-49cc-94fc-73aa7340721d" (UID: "e50e437f-d457-49cc-94fc-73aa7340721d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:39.501377 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.501107    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50e437f-d457-49cc-94fc-73aa7340721d-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:21:39.501377 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.501131    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e50e437f-d457-49cc-94fc-73aa7340721d-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:21:39.501377 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.501157    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e50e437f-d457-49cc-94fc-73aa7340721d-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:21:39.501377 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.501176    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e50e437f-d457-49cc-94fc-73aa7340721d-credential-socket" pod "e50e437f-d457-49cc-94fc-73aa7340721d" (UID: "e50e437f-d457-49cc-94fc-73aa7340721d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:39.501999 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.501973    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e50e437f-d457-49cc-94fc-73aa7340721d-istiod-ca-cert" pod "e50e437f-d457-49cc-94fc-73aa7340721d" (UID: "e50e437f-d457-49cc-94fc-73aa7340721d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:21:39.502694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.502667    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e50e437f-d457-49cc-94fc-73aa7340721d-kube-api-access-8d5mz" pod "e50e437f-d457-49cc-94fc-73aa7340721d" (UID: "e50e437f-d457-49cc-94fc-73aa7340721d"). InnerVolumeSpecName "kube-api-access-8d5mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:21:39.503220 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.503195    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e50e437f-d457-49cc-94fc-73aa7340721d-istio-envoy" pod "e50e437f-d457-49cc-94fc-73aa7340721d" (UID: "e50e437f-d457-49cc-94fc-73aa7340721d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:39.503716 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.503697    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e50e437f-d457-49cc-94fc-73aa7340721d-istio-token" pod "e50e437f-d457-49cc-94fc-73aa7340721d" (UID: "e50e437f-d457-49cc-94fc-73aa7340721d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:21:39.504138 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.504121    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e50e437f-d457-49cc-94fc-73aa7340721d-istio-podinfo" pod "e50e437f-d457-49cc-94fc-73aa7340721d" (UID: "e50e437f-d457-49cc-94fc-73aa7340721d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:21:39.603201 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.603159    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d5mz\" (UniqueName: \"kubernetes.io/projected/e50e437f-d457-49cc-94fc-73aa7340721d-kube-api-access-8d5mz\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:21:39.603201 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.603195    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e50e437f-d457-49cc-94fc-73aa7340721d-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:21:39.603430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.603208    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e50e437f-d457-49cc-94fc-73aa7340721d-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:21:39.603430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.603229    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e50e437f-d457-49cc-94fc-73aa7340721d-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:21:39.603430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.603240    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50e437f-d457-49cc-94fc-73aa7340721d-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:21:39.603430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.603251    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e50e437f-d457-49cc-94fc-73aa7340721d-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:21:39.623267 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.623229    4433 generic.go:356] "Generic (PLEG): container finished" podID="e50e437f-d457-49cc-94fc-73aa7340721d" containerID="cdd767862445376e4927e819feb3ca722099f6b27d26ad271b2f0c9ada783b68" exitCode=0
Oct 30 17:21:39.623473 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.623290    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"
Oct 30 17:21:39.623473 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.623316    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf" event={"ID":"e50e437f-d457-49cc-94fc-73aa7340721d","Type":"ContainerDied","Data":"cdd767862445376e4927e819feb3ca722099f6b27d26ad271b2f0c9ada783b68"}
Oct 30 17:21:39.623473 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.623391    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf" event={"ID":"e50e437f-d457-49cc-94fc-73aa7340721d","Type":"ContainerDied","Data":"45426c46aaadc39bb079393ba688fbf93686f12759079b4f97e6b664112f1554"}
Oct 30 17:21:39.623473 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.623429    4433 scope.go:122] "RemoveContainer" containerID="cdd767862445376e4927e819feb3ca722099f6b27d26ad271b2f0c9ada783b68"
Oct 30 17:21:39.635276 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.635258    4433 scope.go:122] "RemoveContainer" containerID="cdd767862445376e4927e819feb3ca722099f6b27d26ad271b2f0c9ada783b68"
Oct 30 17:21:39.635613 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:21:39.635592    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdd767862445376e4927e819feb3ca722099f6b27d26ad271b2f0c9ada783b68\": container with ID starting with cdd767862445376e4927e819feb3ca722099f6b27d26ad271b2f0c9ada783b68 not found: ID does not exist" containerID="cdd767862445376e4927e819feb3ca722099f6b27d26ad271b2f0c9ada783b68"
Oct 30 17:21:39.635716 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.635619    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd767862445376e4927e819feb3ca722099f6b27d26ad271b2f0c9ada783b68"} err="failed to get container status \"cdd767862445376e4927e819feb3ca722099f6b27d26ad271b2f0c9ada783b68\": rpc error: code = NotFound desc = could not find container \"cdd767862445376e4927e819feb3ca722099f6b27d26ad271b2f0c9ada783b68\": container with ID starting with cdd767862445376e4927e819feb3ca722099f6b27d26ad271b2f0c9ada783b68 not found: ID does not exist"
Oct 30 17:21:39.674779 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.674702    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"]
Oct 30 17:21:39.681671 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.681645    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-2d6sf"]
Oct 30 17:21:39.696147 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:39.696118    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e50e437f-d457-49cc-94fc-73aa7340721d" path="/var/lib/kubelet/pods/e50e437f-d457-49cc-94fc-73aa7340721d/volumes"
Oct 30 17:21:49.586792 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:49.586753    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n"]
Oct 30 17:21:49.587766 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:49.587426    4433 cpu_manager.go:414] "containerMap: removing container" podUID="e50e437f-d457-49cc-94fc-73aa7340721d" containerName="istio-proxy"
Oct 30 17:21:49.587766 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:49.587447    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e50e437f-d457-49cc-94fc-73aa7340721d" containerName="istio-proxy"
Oct 30 17:21:49.587766 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:49.587491    4433 cpu_manager.go:414] "containerMap: removing container" podUID="8f245c01-8039-4ce3-a4de-ce744f348adc" containerName="main"
Oct 30 17:21:49.587766 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:49.587498    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f245c01-8039-4ce3-a4de-ce744f348adc" containerName="main"
Oct 30 17:21:49.587766 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:49.587679    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="e50e437f-d457-49cc-94fc-73aa7340721d" containerName="istio-proxy"
Oct 30 17:21:49.587766 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:49.587699    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f245c01-8039-4ce3-a4de-ce744f348adc" containerName="main"
Oct 30 17:21:49.593640 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:49.593619    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n"
Oct 30 17:21:49.610492 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:49.610464    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n"]
Oct 30 17:21:49.617199 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:49.617162    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-001a96\"/\"openshift-service-ca.crt\""
Oct 30 17:21:49.617967 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:49.617461    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-001a96\"/\"default-dockercfg-nf7ns\""
Oct 30 17:21:49.618138 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:49.617552    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-001a96\"/\"kube-root-ca.crt\""
Oct 30 17:21:49.619450 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:49.619431    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-001a96\"/\"auth-test-model-001a96-kserve-self-signed-certs\""
Oct 30 17:21:49.668181 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:49.668147    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ca3e5af-f7e2-47d5-bc76-66d298b16971-dshm\") pod \"auth-test-model-001a96-kserve-86688f9d4f-8dh2n\" (UID: \"1ca3e5af-f7e2-47d5-bc76-66d298b16971\") " pod="ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n"
Oct 30 17:21:49.668181 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:49.668187    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ca3e5af-f7e2-47d5-bc76-66d298b16971-tmp-dir\") pod \"auth-test-model-001a96-kserve-86688f9d4f-8dh2n\" (UID: \"1ca3e5af-f7e2-47d5-bc76-66d298b16971\") " pod="ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n"
Oct 30 17:21:49.668424 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:49.668213    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ca3e5af-f7e2-47d5-bc76-66d298b16971-tls-certs\") pod \"auth-test-model-001a96-kserve-86688f9d4f-8dh2n\" (UID: \"1ca3e5af-f7e2-47d5-bc76-66d298b16971\") " pod="ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n"
Oct 30 17:21:49.668476 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:49.668458    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqpgb\" (UniqueName: \"kubernetes.io/projected/1ca3e5af-f7e2-47d5-bc76-66d298b16971-kube-api-access-fqpgb\") pod \"auth-test-model-001a96-kserve-86688f9d4f-8dh2n\" (UID: \"1ca3e5af-f7e2-47d5-bc76-66d298b16971\") " pod="ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n"
Oct 30 17:21:49.668536 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:49.668520    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ca3e5af-f7e2-47d5-bc76-66d298b16971-home\") pod \"auth-test-model-001a96-kserve-86688f9d4f-8dh2n\" (UID: \"1ca3e5af-f7e2-47d5-bc76-66d298b16971\") " pod="ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n"
Oct 30 17:21:49.668649 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:49.668632    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ca3e5af-f7e2-47d5-bc76-66d298b16971-model-cache\") pod \"auth-test-model-001a96-kserve-86688f9d4f-8dh2n\" (UID: \"1ca3e5af-f7e2-47d5-bc76-66d298b16971\") " pod="ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n"
Oct 30 17:21:49.770296 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:49.770262    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqpgb\" (UniqueName: \"kubernetes.io/projected/1ca3e5af-f7e2-47d5-bc76-66d298b16971-kube-api-access-fqpgb\") pod \"auth-test-model-001a96-kserve-86688f9d4f-8dh2n\" (UID: \"1ca3e5af-f7e2-47d5-bc76-66d298b16971\") " pod="ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n"
Oct 30 17:21:49.770484 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:49.770332    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ca3e5af-f7e2-47d5-bc76-66d298b16971-home\") pod \"auth-test-model-001a96-kserve-86688f9d4f-8dh2n\" (UID: \"1ca3e5af-f7e2-47d5-bc76-66d298b16971\") " pod="ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n"
Oct 30 17:21:49.770484 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:49.770386    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ca3e5af-f7e2-47d5-bc76-66d298b16971-model-cache\") pod \"auth-test-model-001a96-kserve-86688f9d4f-8dh2n\" (UID: \"1ca3e5af-f7e2-47d5-bc76-66d298b16971\") " pod="ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n"
Oct 30 17:21:49.770484 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:49.770464    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ca3e5af-f7e2-47d5-bc76-66d298b16971-dshm\") pod \"auth-test-model-001a96-kserve-86688f9d4f-8dh2n\" (UID: \"1ca3e5af-f7e2-47d5-bc76-66d298b16971\") " pod="ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n"
Oct 30 17:21:49.770661 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:49.770488    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ca3e5af-f7e2-47d5-bc76-66d298b16971-tmp-dir\") pod \"auth-test-model-001a96-kserve-86688f9d4f-8dh2n\" (UID: \"1ca3e5af-f7e2-47d5-bc76-66d298b16971\") " pod="ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n"
Oct 30 17:21:49.770661 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:49.770524    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ca3e5af-f7e2-47d5-bc76-66d298b16971-tls-certs\") pod \"auth-test-model-001a96-kserve-86688f9d4f-8dh2n\" (UID: \"1ca3e5af-f7e2-47d5-bc76-66d298b16971\") " pod="ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n"
Oct 30 17:21:49.770859 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:49.770826    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ca3e5af-f7e2-47d5-bc76-66d298b16971-home\") pod \"auth-test-model-001a96-kserve-86688f9d4f-8dh2n\" (UID: \"1ca3e5af-f7e2-47d5-bc76-66d298b16971\") " pod="ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n"
Oct 30 17:21:49.771006 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:49.770893    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ca3e5af-f7e2-47d5-bc76-66d298b16971-model-cache\") pod \"auth-test-model-001a96-kserve-86688f9d4f-8dh2n\" (UID: \"1ca3e5af-f7e2-47d5-bc76-66d298b16971\") " pod="ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n"
Oct 30 17:21:49.771006 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:49.770949    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ca3e5af-f7e2-47d5-bc76-66d298b16971-tmp-dir\") pod \"auth-test-model-001a96-kserve-86688f9d4f-8dh2n\" (UID: \"1ca3e5af-f7e2-47d5-bc76-66d298b16971\") " pod="ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n"
Oct 30 17:21:49.772894 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:49.772869    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ca3e5af-f7e2-47d5-bc76-66d298b16971-dshm\") pod \"auth-test-model-001a96-kserve-86688f9d4f-8dh2n\" (UID: \"1ca3e5af-f7e2-47d5-bc76-66d298b16971\") " pod="ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n"
Oct 30 17:21:49.773187 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:49.773154    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ca3e5af-f7e2-47d5-bc76-66d298b16971-tls-certs\") pod \"auth-test-model-001a96-kserve-86688f9d4f-8dh2n\" (UID: \"1ca3e5af-f7e2-47d5-bc76-66d298b16971\") " pod="ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n"
Oct 30 17:21:49.794201 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:49.794160    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqpgb\" (UniqueName: \"kubernetes.io/projected/1ca3e5af-f7e2-47d5-bc76-66d298b16971-kube-api-access-fqpgb\") pod \"auth-test-model-001a96-kserve-86688f9d4f-8dh2n\" (UID: \"1ca3e5af-f7e2-47d5-bc76-66d298b16971\") " pod="ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n"
Oct 30 17:21:49.922480 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:49.918648    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n"
Oct 30 17:21:50.287874 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:50.287839    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n"]
Oct 30 17:21:50.289699 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:21:50.289674    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ca3e5af_f7e2_47d5_bc76_66d298b16971.slice/crio-9f8fdf8bbcc21cbf7407c8a3bfd15d71308660962e6d07da10270e3fa177ebb1 WatchSource:0}: Error finding container 9f8fdf8bbcc21cbf7407c8a3bfd15d71308660962e6d07da10270e3fa177ebb1: Status 404 returned error can't find the container with id 9f8fdf8bbcc21cbf7407c8a3bfd15d71308660962e6d07da10270e3fa177ebb1
Oct 30 17:21:50.702620 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:50.702579    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n" event={"ID":"1ca3e5af-f7e2-47d5-bc76-66d298b16971","Type":"ContainerStarted","Data":"093262a71277c7703fe3168c5cdb17d6183bef9017575a7319cfb4bc8a72ae87"}
Oct 30 17:21:50.703014 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:50.702634    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n" event={"ID":"1ca3e5af-f7e2-47d5-bc76-66d298b16971","Type":"ContainerStarted","Data":"9f8fdf8bbcc21cbf7407c8a3bfd15d71308660962e6d07da10270e3fa177ebb1"}
Oct 30 17:21:50.745599 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:50.745546    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n" podStartSLOduration=1.745531905 podStartE2EDuration="1.745531905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:21:50.743457945 +0000 UTC m=+2087.178360147" watchObservedRunningTime="2026-10-30 17:21:50.745531905 +0000 UTC m=+2087.180434109"
Oct 30 17:21:50.919546 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:50.919509    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n"
Oct 30 17:21:56.127429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:56.127384    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"]
Oct 30 17:21:56.127883 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:56.127609    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz" podUID="7dded80b-9e71-4566-944c-4d548c59a684" containerName="istio-proxy" containerID="cri-o://13e25913a9461eb4e01fabbd28fa4000ee85fb8b7c9d7ed13b15baa9f97396eb" gracePeriod=30
Oct 30 17:21:56.978730 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:21:56.978669    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:21:59.919359 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:59.919313    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n"
Oct 30 17:21:59.932118 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:21:59.932088    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n"
Oct 30 17:22:00.436462 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:00.436383    4433 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:22:00.436643 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:00.436477    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz" podUID="7dded80b-9e71-4566-944c-4d548c59a684" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:22:00.787945 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:00.787851    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n"
Oct 30 17:22:01.410042 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.409963    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"
Oct 30 17:22:01.476236 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.476194    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dded80b-9e71-4566-944c-4d548c59a684-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dded80b-9e71-4566-944c-4d548c59a684-workload-socket\") pod \"7dded80b-9e71-4566-944c-4d548c59a684\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") "
Oct 30 17:22:01.476431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.476314    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dded80b-9e71-4566-944c-4d548c59a684-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dded80b-9e71-4566-944c-4d548c59a684-workload-certs\") pod \"7dded80b-9e71-4566-944c-4d548c59a684\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") "
Oct 30 17:22:01.476431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.476356    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dded80b-9e71-4566-944c-4d548c59a684-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dded80b-9e71-4566-944c-4d548c59a684-istio-data\") pod \"7dded80b-9e71-4566-944c-4d548c59a684\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") "
Oct 30 17:22:01.476431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.476384    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dded80b-9e71-4566-944c-4d548c59a684-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dded80b-9e71-4566-944c-4d548c59a684-credential-socket\") pod \"7dded80b-9e71-4566-944c-4d548c59a684\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") "
Oct 30 17:22:01.476578 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.476472    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dded80b-9e71-4566-944c-4d548c59a684-kube-api-access-rvmft\" (UniqueName: \"kubernetes.io/projected/7dded80b-9e71-4566-944c-4d548c59a684-kube-api-access-rvmft\") pod \"7dded80b-9e71-4566-944c-4d548c59a684\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") "
Oct 30 17:22:01.476578 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.476517    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dded80b-9e71-4566-944c-4d548c59a684-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dded80b-9e71-4566-944c-4d548c59a684-istiod-ca-cert\") pod \"7dded80b-9e71-4566-944c-4d548c59a684\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") "
Oct 30 17:22:01.476578 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.476526    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dded80b-9e71-4566-944c-4d548c59a684-workload-socket" pod "7dded80b-9e71-4566-944c-4d548c59a684" (UID: "7dded80b-9e71-4566-944c-4d548c59a684"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:22:01.476722 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.476688    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dded80b-9e71-4566-944c-4d548c59a684-istio-token\" (UniqueName: \"kubernetes.io/projected/7dded80b-9e71-4566-944c-4d548c59a684-istio-token\") pod \"7dded80b-9e71-4566-944c-4d548c59a684\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") "
Oct 30 17:22:01.476722 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.476696    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dded80b-9e71-4566-944c-4d548c59a684-workload-certs" pod "7dded80b-9e71-4566-944c-4d548c59a684" (UID: "7dded80b-9e71-4566-944c-4d548c59a684"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:22:01.476722 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.476696    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dded80b-9e71-4566-944c-4d548c59a684-istio-data" pod "7dded80b-9e71-4566-944c-4d548c59a684" (UID: "7dded80b-9e71-4566-944c-4d548c59a684"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:22:01.476869 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.476765    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dded80b-9e71-4566-944c-4d548c59a684-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dded80b-9e71-4566-944c-4d548c59a684-istio-envoy\") pod \"7dded80b-9e71-4566-944c-4d548c59a684\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") "
Oct 30 17:22:01.476869 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.476785    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dded80b-9e71-4566-944c-4d548c59a684-credential-socket" pod "7dded80b-9e71-4566-944c-4d548c59a684" (UID: "7dded80b-9e71-4566-944c-4d548c59a684"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:22:01.476869 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.476803    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7dded80b-9e71-4566-944c-4d548c59a684-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dded80b-9e71-4566-944c-4d548c59a684-istio-podinfo\") pod \"7dded80b-9e71-4566-944c-4d548c59a684\" (UID: \"7dded80b-9e71-4566-944c-4d548c59a684\") "
Oct 30 17:22:01.477057 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.477031    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dded80b-9e71-4566-944c-4d548c59a684-istiod-ca-cert" pod "7dded80b-9e71-4566-944c-4d548c59a684" (UID: "7dded80b-9e71-4566-944c-4d548c59a684"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:22:01.477874 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.477840    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dded80b-9e71-4566-944c-4d548c59a684-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:01.478031 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.478002    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dded80b-9e71-4566-944c-4d548c59a684-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:01.478031 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.478023    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dded80b-9e71-4566-944c-4d548c59a684-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:01.478151 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.478037    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dded80b-9e71-4566-944c-4d548c59a684-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:01.478151 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.478052    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dded80b-9e71-4566-944c-4d548c59a684-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:01.479154 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.479123    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dded80b-9e71-4566-944c-4d548c59a684-istio-token" pod "7dded80b-9e71-4566-944c-4d548c59a684" (UID: "7dded80b-9e71-4566-944c-4d548c59a684"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:22:01.479254 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.479214    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7dded80b-9e71-4566-944c-4d548c59a684-istio-podinfo" pod "7dded80b-9e71-4566-944c-4d548c59a684" (UID: "7dded80b-9e71-4566-944c-4d548c59a684"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:22:01.479396 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.479358    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dded80b-9e71-4566-944c-4d548c59a684-istio-envoy" pod "7dded80b-9e71-4566-944c-4d548c59a684" (UID: "7dded80b-9e71-4566-944c-4d548c59a684"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:22:01.480623 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.480601    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dded80b-9e71-4566-944c-4d548c59a684-kube-api-access-rvmft" pod "7dded80b-9e71-4566-944c-4d548c59a684" (UID: "7dded80b-9e71-4566-944c-4d548c59a684"). InnerVolumeSpecName "kube-api-access-rvmft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:22:01.579726 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.579676    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dded80b-9e71-4566-944c-4d548c59a684-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:01.579726 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.579713    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dded80b-9e71-4566-944c-4d548c59a684-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:01.579726 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.579727    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dded80b-9e71-4566-944c-4d548c59a684-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:01.580095 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.579740    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvmft\" (UniqueName: \"kubernetes.io/projected/7dded80b-9e71-4566-944c-4d548c59a684-kube-api-access-rvmft\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:01.786355 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.786308    4433 generic.go:356] "Generic (PLEG): container finished" podID="7dded80b-9e71-4566-944c-4d548c59a684" containerID="13e25913a9461eb4e01fabbd28fa4000ee85fb8b7c9d7ed13b15baa9f97396eb" exitCode=0
Oct 30 17:22:01.786355 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.786352    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"
Oct 30 17:22:01.786564 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.786390    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz" event={"ID":"7dded80b-9e71-4566-944c-4d548c59a684","Type":"ContainerDied","Data":"13e25913a9461eb4e01fabbd28fa4000ee85fb8b7c9d7ed13b15baa9f97396eb"}
Oct 30 17:22:01.786564 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.786479    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz" event={"ID":"7dded80b-9e71-4566-944c-4d548c59a684","Type":"ContainerDied","Data":"18210d37ee2d6df7460d2fabaacbe0a7ab1b0846ccbe693ef2fda383d5a1ad2f"}
Oct 30 17:22:01.786564 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.786519    4433 scope.go:122] "RemoveContainer" containerID="13e25913a9461eb4e01fabbd28fa4000ee85fb8b7c9d7ed13b15baa9f97396eb"
Oct 30 17:22:01.796639 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.796615    4433 scope.go:122] "RemoveContainer" containerID="13e25913a9461eb4e01fabbd28fa4000ee85fb8b7c9d7ed13b15baa9f97396eb"
Oct 30 17:22:01.796952 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:22:01.796931    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13e25913a9461eb4e01fabbd28fa4000ee85fb8b7c9d7ed13b15baa9f97396eb\": container with ID starting with 13e25913a9461eb4e01fabbd28fa4000ee85fb8b7c9d7ed13b15baa9f97396eb not found: ID does not exist" containerID="13e25913a9461eb4e01fabbd28fa4000ee85fb8b7c9d7ed13b15baa9f97396eb"
Oct 30 17:22:01.797014 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.796967    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13e25913a9461eb4e01fabbd28fa4000ee85fb8b7c9d7ed13b15baa9f97396eb"} err="failed to get container status \"13e25913a9461eb4e01fabbd28fa4000ee85fb8b7c9d7ed13b15baa9f97396eb\": rpc error: code = NotFound desc = could not find container \"13e25913a9461eb4e01fabbd28fa4000ee85fb8b7c9d7ed13b15baa9f97396eb\": container with ID starting with 13e25913a9461eb4e01fabbd28fa4000ee85fb8b7c9d7ed13b15baa9f97396eb not found: ID does not exist"
Oct 30 17:22:01.841022 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.840992    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"]
Oct 30 17:22:01.852289 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:01.852261    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-vpktz"]
Oct 30 17:22:03.697982 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:03.697948    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dded80b-9e71-4566-944c-4d548c59a684" path="/var/lib/kubelet/pods/7dded80b-9e71-4566-944c-4d548c59a684/volumes"
Oct 30 17:22:04.857502 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:04.857467    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64"]
Oct 30 17:22:13.909566 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:13.909528    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"]
Oct 30 17:22:13.910285 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:13.910246    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5" podUID="c07529e2-9bcd-4ec9-9c98-e0bb48c8effb" containerName="istio-proxy" containerID="cri-o://77dcb683431ee8eaaaa39784b6d93f534818ac8768b96608a977d28390f7032f" gracePeriod=30
Oct 30 17:22:13.916129 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:13.916071    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"]
Oct 30 17:22:13.916319 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:13.916291    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx" podUID="bb49dfad-1e85-4b96-9dd8-42374a708643" containerName="istio-proxy" containerID="cri-o://32eba274730c1cae86773f4ec9032df5676779149c5b8b78ece6e0af3deff139" gracePeriod=30
Oct 30 17:22:14.410156 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:14.410121    4433 patch_prober.go:29] interesting pod/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:22:14.410319 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:14.410185    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5" podUID="c07529e2-9bcd-4ec9-9c98-e0bb48c8effb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:22:14.569991 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:14.569952    4433 patch_prober.go:29] interesting pod/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:22:14.570147 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:14.570010    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx" podUID="bb49dfad-1e85-4b96-9dd8-42374a708643" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:22:19.204325 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.204256    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"
Oct 30 17:22:19.246238 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.246209    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"
Oct 30 17:22:19.259474 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.259451    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb49dfad-1e85-4b96-9dd8-42374a708643-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb49dfad-1e85-4b96-9dd8-42374a708643-istio-envoy\") pod \"bb49dfad-1e85-4b96-9dd8-42374a708643\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") "
Oct 30 17:22:19.259619 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.259500    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb49dfad-1e85-4b96-9dd8-42374a708643-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb49dfad-1e85-4b96-9dd8-42374a708643-credential-socket\") pod \"bb49dfad-1e85-4b96-9dd8-42374a708643\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") "
Oct 30 17:22:19.259619 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.259592    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb49dfad-1e85-4b96-9dd8-42374a708643-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb49dfad-1e85-4b96-9dd8-42374a708643-istio-data\") pod \"bb49dfad-1e85-4b96-9dd8-42374a708643\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") "
Oct 30 17:22:19.259737 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.259674    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb49dfad-1e85-4b96-9dd8-42374a708643-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb49dfad-1e85-4b96-9dd8-42374a708643-workload-certs\") pod \"bb49dfad-1e85-4b96-9dd8-42374a708643\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") "
Oct 30 17:22:19.259737 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.259719    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb49dfad-1e85-4b96-9dd8-42374a708643-istio-token\" (UniqueName: \"kubernetes.io/projected/bb49dfad-1e85-4b96-9dd8-42374a708643-istio-token\") pod \"bb49dfad-1e85-4b96-9dd8-42374a708643\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") "
Oct 30 17:22:19.259919 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.259882    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb49dfad-1e85-4b96-9dd8-42374a708643-kube-api-access-f6w54\" (UniqueName: \"kubernetes.io/projected/bb49dfad-1e85-4b96-9dd8-42374a708643-kube-api-access-f6w54\") pod \"bb49dfad-1e85-4b96-9dd8-42374a708643\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") "
Oct 30 17:22:19.260041 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.259971    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb49dfad-1e85-4b96-9dd8-42374a708643-credential-socket" pod "bb49dfad-1e85-4b96-9dd8-42374a708643" (UID: "bb49dfad-1e85-4b96-9dd8-42374a708643"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:22:19.260041 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.259981    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb49dfad-1e85-4b96-9dd8-42374a708643-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb49dfad-1e85-4b96-9dd8-42374a708643-istiod-ca-cert\") pod \"bb49dfad-1e85-4b96-9dd8-42374a708643\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") "
Oct 30 17:22:19.260214 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.260152    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb49dfad-1e85-4b96-9dd8-42374a708643-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb49dfad-1e85-4b96-9dd8-42374a708643-istio-podinfo\") pod \"bb49dfad-1e85-4b96-9dd8-42374a708643\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") "
Oct 30 17:22:19.260214 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.260151    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb49dfad-1e85-4b96-9dd8-42374a708643-istio-data" pod "bb49dfad-1e85-4b96-9dd8-42374a708643" (UID: "bb49dfad-1e85-4b96-9dd8-42374a708643"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:22:19.260356 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.260253    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb49dfad-1e85-4b96-9dd8-42374a708643-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb49dfad-1e85-4b96-9dd8-42374a708643-workload-socket\") pod \"bb49dfad-1e85-4b96-9dd8-42374a708643\" (UID: \"bb49dfad-1e85-4b96-9dd8-42374a708643\") "
Oct 30 17:22:19.260356 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.260291    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb49dfad-1e85-4b96-9dd8-42374a708643-istiod-ca-cert" pod "bb49dfad-1e85-4b96-9dd8-42374a708643" (UID: "bb49dfad-1e85-4b96-9dd8-42374a708643"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:22:19.260844 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.260602    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb49dfad-1e85-4b96-9dd8-42374a708643-workload-certs" pod "bb49dfad-1e85-4b96-9dd8-42374a708643" (UID: "bb49dfad-1e85-4b96-9dd8-42374a708643"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:22:19.260844 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.260816    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb49dfad-1e85-4b96-9dd8-42374a708643-workload-socket" pod "bb49dfad-1e85-4b96-9dd8-42374a708643" (UID: "bb49dfad-1e85-4b96-9dd8-42374a708643"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:22:19.261023 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.260997    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb49dfad-1e85-4b96-9dd8-42374a708643-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:19.261095 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.261031    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb49dfad-1e85-4b96-9dd8-42374a708643-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:19.261095 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.261044    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb49dfad-1e85-4b96-9dd8-42374a708643-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:19.261095 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.261060    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb49dfad-1e85-4b96-9dd8-42374a708643-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:19.261095 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.261073    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb49dfad-1e85-4b96-9dd8-42374a708643-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:19.262700 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.262669    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb49dfad-1e85-4b96-9dd8-42374a708643-istio-token" pod "bb49dfad-1e85-4b96-9dd8-42374a708643" (UID: "bb49dfad-1e85-4b96-9dd8-42374a708643"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:22:19.262854 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.262830    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb49dfad-1e85-4b96-9dd8-42374a708643-istio-envoy" pod "bb49dfad-1e85-4b96-9dd8-42374a708643" (UID: "bb49dfad-1e85-4b96-9dd8-42374a708643"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:22:19.263077 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.263059    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb49dfad-1e85-4b96-9dd8-42374a708643-kube-api-access-f6w54" pod "bb49dfad-1e85-4b96-9dd8-42374a708643" (UID: "bb49dfad-1e85-4b96-9dd8-42374a708643"). InnerVolumeSpecName "kube-api-access-f6w54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:22:19.263182 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.263165    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb49dfad-1e85-4b96-9dd8-42374a708643-istio-podinfo" pod "bb49dfad-1e85-4b96-9dd8-42374a708643" (UID: "bb49dfad-1e85-4b96-9dd8-42374a708643"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:22:19.361792 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.361761    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-kube-api-access-4jrqm\" (UniqueName: \"kubernetes.io/projected/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-kube-api-access-4jrqm\") pod \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") "
Oct 30 17:22:19.361984 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.361826    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-istiod-ca-cert\") pod \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") "
Oct 30 17:22:19.361984 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.361864    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-credential-socket\") pod \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") "
Oct 30 17:22:19.361984 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.361917    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-istio-data\") pod \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") "
Oct 30 17:22:19.362150 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.362007    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-istio-podinfo\") pod \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") "
Oct 30 17:22:19.362150 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.362058    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-istio-envoy\") pod \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") "
Oct 30 17:22:19.362150 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.362097    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-workload-certs\") pod \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") "
Oct 30 17:22:19.362383 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.362178    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-workload-socket\") pod \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") "
Oct 30 17:22:19.362383 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.362224    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-credential-socket" pod "c07529e2-9bcd-4ec9-9c98-e0bb48c8effb" (UID: "c07529e2-9bcd-4ec9-9c98-e0bb48c8effb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:22:19.362383 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.362264    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-istiod-ca-cert" pod "c07529e2-9bcd-4ec9-9c98-e0bb48c8effb" (UID: "c07529e2-9bcd-4ec9-9c98-e0bb48c8effb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:22:19.362383 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.362300    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-istio-data" pod "c07529e2-9bcd-4ec9-9c98-e0bb48c8effb" (UID: "c07529e2-9bcd-4ec9-9c98-e0bb48c8effb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:22:19.362383 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.362329    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-istio-token\" (UniqueName: \"kubernetes.io/projected/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-istio-token\") pod \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\" (UID: \"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb\") "
Oct 30 17:22:19.362648 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.362492    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-workload-socket" pod "c07529e2-9bcd-4ec9-9c98-e0bb48c8effb" (UID: "c07529e2-9bcd-4ec9-9c98-e0bb48c8effb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:22:19.362648 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.362600    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-workload-certs" pod "c07529e2-9bcd-4ec9-9c98-e0bb48c8effb" (UID: "c07529e2-9bcd-4ec9-9c98-e0bb48c8effb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:22:19.363113 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.363088    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:19.363213 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.363118    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb49dfad-1e85-4b96-9dd8-42374a708643-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:19.363213 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.363134    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6w54\" (UniqueName: \"kubernetes.io/projected/bb49dfad-1e85-4b96-9dd8-42374a708643-kube-api-access-f6w54\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:19.363213 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.363150    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:19.363213 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.363166    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:19.363213 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.363182    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:19.363213 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.363194    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb49dfad-1e85-4b96-9dd8-42374a708643-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:19.363213 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.363207    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb49dfad-1e85-4b96-9dd8-42374a708643-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:19.363626 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.363219    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:19.364247 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.364223    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-istio-envoy" pod "c07529e2-9bcd-4ec9-9c98-e0bb48c8effb" (UID: "c07529e2-9bcd-4ec9-9c98-e0bb48c8effb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:22:19.364469 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.364445    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-istio-podinfo" pod "c07529e2-9bcd-4ec9-9c98-e0bb48c8effb" (UID: "c07529e2-9bcd-4ec9-9c98-e0bb48c8effb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:22:19.364553 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.364478    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-kube-api-access-4jrqm" pod "c07529e2-9bcd-4ec9-9c98-e0bb48c8effb" (UID: "c07529e2-9bcd-4ec9-9c98-e0bb48c8effb"). InnerVolumeSpecName "kube-api-access-4jrqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:22:19.364629 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.364611    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-istio-token" pod "c07529e2-9bcd-4ec9-9c98-e0bb48c8effb" (UID: "c07529e2-9bcd-4ec9-9c98-e0bb48c8effb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:22:19.464606 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.464530    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:19.464606 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.464560    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:19.464606 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.464570    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:19.464606 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.464578    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jrqm\" (UniqueName: \"kubernetes.io/projected/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb-kube-api-access-4jrqm\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:19.885726 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.885632    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64" podUID="3f6eaf54-07a2-4010-9b79-e82670f15262" containerName="main" containerID="cri-o://09fb7757e785afe73e877c523ea1d706aec5c7764fba9c6fd71e5d622123d9ff" gracePeriod=45
Oct 30 17:22:19.916267 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.916233    4433 generic.go:356] "Generic (PLEG): container finished" podID="c07529e2-9bcd-4ec9-9c98-e0bb48c8effb" containerID="77dcb683431ee8eaaaa39784b6d93f534818ac8768b96608a977d28390f7032f" exitCode=0
Oct 30 17:22:19.916437 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.916281    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"
Oct 30 17:22:19.916437 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.916294    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5" event={"ID":"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb","Type":"ContainerDied","Data":"77dcb683431ee8eaaaa39784b6d93f534818ac8768b96608a977d28390f7032f"}
Oct 30 17:22:19.916437 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.916349    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5" event={"ID":"c07529e2-9bcd-4ec9-9c98-e0bb48c8effb","Type":"ContainerDied","Data":"e3d79ec5ee458c5aec08d6d962f2db21858aef475734cf11f09e5db2939002dd"}
Oct 30 17:22:19.916437 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.916371    4433 scope.go:122] "RemoveContainer" containerID="77dcb683431ee8eaaaa39784b6d93f534818ac8768b96608a977d28390f7032f"
Oct 30 17:22:19.917769 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.917745    4433 generic.go:356] "Generic (PLEG): container finished" podID="bb49dfad-1e85-4b96-9dd8-42374a708643" containerID="32eba274730c1cae86773f4ec9032df5676779149c5b8b78ece6e0af3deff139" exitCode=0
Oct 30 17:22:19.917827 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.917788    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"
Oct 30 17:22:19.917827 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.917793    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx" event={"ID":"bb49dfad-1e85-4b96-9dd8-42374a708643","Type":"ContainerDied","Data":"32eba274730c1cae86773f4ec9032df5676779149c5b8b78ece6e0af3deff139"}
Oct 30 17:22:19.917902 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.917831    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx" event={"ID":"bb49dfad-1e85-4b96-9dd8-42374a708643","Type":"ContainerDied","Data":"b5905610ee7ed00d35449863f8a56e44d469d5d01d98a52b5be5368c4dcf6967"}
Oct 30 17:22:19.926821 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.926792    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2"]
Oct 30 17:22:19.927048 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:19.927018    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2" podUID="fc12f255-74a5-4c88-b67b-a0c377797e84" containerName="maas-api" containerID="cri-o://f14b17f8f6814cc9e0771490a8163284320bf592c1137a85c50f5c565e6cf340" gracePeriod=30
Oct 30 17:22:20.021656 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.021599    4433 scope.go:122] "RemoveContainer" containerID="77dcb683431ee8eaaaa39784b6d93f534818ac8768b96608a977d28390f7032f"
Oct 30 17:22:20.022048 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:22:20.022018    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77dcb683431ee8eaaaa39784b6d93f534818ac8768b96608a977d28390f7032f\": container with ID starting with 77dcb683431ee8eaaaa39784b6d93f534818ac8768b96608a977d28390f7032f not found: ID does not exist" containerID="77dcb683431ee8eaaaa39784b6d93f534818ac8768b96608a977d28390f7032f"
Oct 30 17:22:20.022165 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.022062    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77dcb683431ee8eaaaa39784b6d93f534818ac8768b96608a977d28390f7032f"} err="failed to get container status \"77dcb683431ee8eaaaa39784b6d93f534818ac8768b96608a977d28390f7032f\": rpc error: code = NotFound desc = could not find container \"77dcb683431ee8eaaaa39784b6d93f534818ac8768b96608a977d28390f7032f\": container with ID starting with 77dcb683431ee8eaaaa39784b6d93f534818ac8768b96608a977d28390f7032f not found: ID does not exist"
Oct 30 17:22:20.022165 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.022087    4433 scope.go:122] "RemoveContainer" containerID="32eba274730c1cae86773f4ec9032df5676779149c5b8b78ece6e0af3deff139"
Oct 30 17:22:20.045812 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.044786    4433 scope.go:122] "RemoveContainer" containerID="32eba274730c1cae86773f4ec9032df5676779149c5b8b78ece6e0af3deff139"
Oct 30 17:22:20.045812 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:22:20.045170    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32eba274730c1cae86773f4ec9032df5676779149c5b8b78ece6e0af3deff139\": container with ID starting with 32eba274730c1cae86773f4ec9032df5676779149c5b8b78ece6e0af3deff139 not found: ID does not exist" containerID="32eba274730c1cae86773f4ec9032df5676779149c5b8b78ece6e0af3deff139"
Oct 30 17:22:20.045812 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.045208    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32eba274730c1cae86773f4ec9032df5676779149c5b8b78ece6e0af3deff139"} err="failed to get container status \"32eba274730c1cae86773f4ec9032df5676779149c5b8b78ece6e0af3deff139\": rpc error: code = NotFound desc = could not find container \"32eba274730c1cae86773f4ec9032df5676779149c5b8b78ece6e0af3deff139\": container with ID starting with 32eba274730c1cae86773f4ec9032df5676779149c5b8b78ece6e0af3deff139 not found: ID does not exist"
Oct 30 17:22:20.061006 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.060134    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"]
Oct 30 17:22:20.066970 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.066933    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-vlvfx"]
Oct 30 17:22:20.121465 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.121431    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"]
Oct 30 17:22:20.126468 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.126441    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-f5xg5"]
Oct 30 17:22:20.584682 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.584649    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64"
Oct 30 17:22:20.630856 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.630830    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2"
Oct 30 17:22:20.649345 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.649299    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f6eaf54-07a2-4010-9b79-e82670f15262-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f6eaf54-07a2-4010-9b79-e82670f15262-model-cache\") pod \"3f6eaf54-07a2-4010-9b79-e82670f15262\" (UID: \"3f6eaf54-07a2-4010-9b79-e82670f15262\") "
Oct 30 17:22:20.649651 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.649357    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f6eaf54-07a2-4010-9b79-e82670f15262-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f6eaf54-07a2-4010-9b79-e82670f15262-tmp-dir\") pod \"3f6eaf54-07a2-4010-9b79-e82670f15262\" (UID: \"3f6eaf54-07a2-4010-9b79-e82670f15262\") "
Oct 30 17:22:20.649651 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.649491    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f6eaf54-07a2-4010-9b79-e82670f15262-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f6eaf54-07a2-4010-9b79-e82670f15262-dshm\") pod \"3f6eaf54-07a2-4010-9b79-e82670f15262\" (UID: \"3f6eaf54-07a2-4010-9b79-e82670f15262\") "
Oct 30 17:22:20.649651 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.649549    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc12f255-74a5-4c88-b67b-a0c377797e84-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc12f255-74a5-4c88-b67b-a0c377797e84-metrics-certs\") pod \"fc12f255-74a5-4c88-b67b-a0c377797e84\" (UID: \"fc12f255-74a5-4c88-b67b-a0c377797e84\") "
Oct 30 17:22:20.649651 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.649580    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc12f255-74a5-4c88-b67b-a0c377797e84-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc12f255-74a5-4c88-b67b-a0c377797e84-openshift-service-ca\") pod \"fc12f255-74a5-4c88-b67b-a0c377797e84\" (UID: \"fc12f255-74a5-4c88-b67b-a0c377797e84\") "
Oct 30 17:22:20.649651 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.649618    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f6eaf54-07a2-4010-9b79-e82670f15262-home\" (UniqueName: \"kubernetes.io/empty-dir/3f6eaf54-07a2-4010-9b79-e82670f15262-home\") pod \"3f6eaf54-07a2-4010-9b79-e82670f15262\" (UID: \"3f6eaf54-07a2-4010-9b79-e82670f15262\") "
Oct 30 17:22:20.649651 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.649653    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f6eaf54-07a2-4010-9b79-e82670f15262-kube-api-access-7pbc4\" (UniqueName: \"kubernetes.io/projected/3f6eaf54-07a2-4010-9b79-e82670f15262-kube-api-access-7pbc4\") pod \"3f6eaf54-07a2-4010-9b79-e82670f15262\" (UID: \"3f6eaf54-07a2-4010-9b79-e82670f15262\") "
Oct 30 17:22:20.649973 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.649687    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc12f255-74a5-4c88-b67b-a0c377797e84-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc12f255-74a5-4c88-b67b-a0c377797e84-maas-api-tls\") pod \"fc12f255-74a5-4c88-b67b-a0c377797e84\" (UID: \"fc12f255-74a5-4c88-b67b-a0c377797e84\") "
Oct 30 17:22:20.649973 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.649709    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f6eaf54-07a2-4010-9b79-e82670f15262-tmp-dir" pod "3f6eaf54-07a2-4010-9b79-e82670f15262" (UID: "3f6eaf54-07a2-4010-9b79-e82670f15262"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:22:20.649973 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.649726    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f6eaf54-07a2-4010-9b79-e82670f15262-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f6eaf54-07a2-4010-9b79-e82670f15262-tls-certs\") pod \"3f6eaf54-07a2-4010-9b79-e82670f15262\" (UID: \"3f6eaf54-07a2-4010-9b79-e82670f15262\") "
Oct 30 17:22:20.649973 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.649860    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc12f255-74a5-4c88-b67b-a0c377797e84-kube-api-access-xfpq2\" (UniqueName: \"kubernetes.io/projected/fc12f255-74a5-4c88-b67b-a0c377797e84-kube-api-access-xfpq2\") pod \"fc12f255-74a5-4c88-b67b-a0c377797e84\" (UID: \"fc12f255-74a5-4c88-b67b-a0c377797e84\") "
Oct 30 17:22:20.650197 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.650173    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc12f255-74a5-4c88-b67b-a0c377797e84-openshift-service-ca" pod "fc12f255-74a5-4c88-b67b-a0c377797e84" (UID: "fc12f255-74a5-4c88-b67b-a0c377797e84"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:22:20.650469 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.650448    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f6eaf54-07a2-4010-9b79-e82670f15262-home" pod "3f6eaf54-07a2-4010-9b79-e82670f15262" (UID: "3f6eaf54-07a2-4010-9b79-e82670f15262"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:22:20.650817 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.650791    4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f6eaf54-07a2-4010-9b79-e82670f15262-tmp-dir\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:20.650909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.650823    4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc12f255-74a5-4c88-b67b-a0c377797e84-openshift-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:20.650909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.650838    4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f6eaf54-07a2-4010-9b79-e82670f15262-home\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:20.651542 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.651517    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f6eaf54-07a2-4010-9b79-e82670f15262-model-cache" pod "3f6eaf54-07a2-4010-9b79-e82670f15262" (UID: "3f6eaf54-07a2-4010-9b79-e82670f15262"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:22:20.653240 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.653213    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f6eaf54-07a2-4010-9b79-e82670f15262-tls-certs" pod "3f6eaf54-07a2-4010-9b79-e82670f15262" (UID: "3f6eaf54-07a2-4010-9b79-e82670f15262"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:22:20.653240 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.653234    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc12f255-74a5-4c88-b67b-a0c377797e84-kube-api-access-xfpq2" pod "fc12f255-74a5-4c88-b67b-a0c377797e84" (UID: "fc12f255-74a5-4c88-b67b-a0c377797e84"). InnerVolumeSpecName "kube-api-access-xfpq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:22:20.653427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.653218    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f6eaf54-07a2-4010-9b79-e82670f15262-kube-api-access-7pbc4" pod "3f6eaf54-07a2-4010-9b79-e82670f15262" (UID: "3f6eaf54-07a2-4010-9b79-e82670f15262"). InnerVolumeSpecName "kube-api-access-7pbc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:22:20.653647 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.653615    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc12f255-74a5-4c88-b67b-a0c377797e84-maas-api-tls" pod "fc12f255-74a5-4c88-b67b-a0c377797e84" (UID: "fc12f255-74a5-4c88-b67b-a0c377797e84"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:22:20.654884 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.654860    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f6eaf54-07a2-4010-9b79-e82670f15262-dshm" pod "3f6eaf54-07a2-4010-9b79-e82670f15262" (UID: "3f6eaf54-07a2-4010-9b79-e82670f15262"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:22:20.655273 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.655252    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc12f255-74a5-4c88-b67b-a0c377797e84-metrics-certs" pod "fc12f255-74a5-4c88-b67b-a0c377797e84" (UID: "fc12f255-74a5-4c88-b67b-a0c377797e84"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:22:20.751720 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.751688    4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f6eaf54-07a2-4010-9b79-e82670f15262-dshm\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:20.751720 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.751716    4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc12f255-74a5-4c88-b67b-a0c377797e84-metrics-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:20.751720 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.751727    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pbc4\" (UniqueName: \"kubernetes.io/projected/3f6eaf54-07a2-4010-9b79-e82670f15262-kube-api-access-7pbc4\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:20.752013 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.751735    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc12f255-74a5-4c88-b67b-a0c377797e84-maas-api-tls\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:20.752013 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.751743    4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f6eaf54-07a2-4010-9b79-e82670f15262-tls-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:20.752013 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.751751    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfpq2\" (UniqueName: \"kubernetes.io/projected/fc12f255-74a5-4c88-b67b-a0c377797e84-kube-api-access-xfpq2\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:20.752013 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.751758    4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f6eaf54-07a2-4010-9b79-e82670f15262-model-cache\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:20.926110 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.926075    4433 generic.go:356] "Generic (PLEG): container finished" podID="fc12f255-74a5-4c88-b67b-a0c377797e84" containerID="f14b17f8f6814cc9e0771490a8163284320bf592c1137a85c50f5c565e6cf340" exitCode=0
Oct 30 17:22:20.926292 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.926127    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2"
Oct 30 17:22:20.926292 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.926164    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2" event={"ID":"fc12f255-74a5-4c88-b67b-a0c377797e84","Type":"ContainerDied","Data":"f14b17f8f6814cc9e0771490a8163284320bf592c1137a85c50f5c565e6cf340"}
Oct 30 17:22:20.926292 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.926232    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2" event={"ID":"fc12f255-74a5-4c88-b67b-a0c377797e84","Type":"ContainerDied","Data":"6835d785494154bae86a6d5dac4714c4f139266e037af579dcc336a14b54da67"}
Oct 30 17:22:20.926292 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.926261    4433 scope.go:122] "RemoveContainer" containerID="f14b17f8f6814cc9e0771490a8163284320bf592c1137a85c50f5c565e6cf340"
Oct 30 17:22:20.928744 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.928714    4433 generic.go:356] "Generic (PLEG): container finished" podID="3f6eaf54-07a2-4010-9b79-e82670f15262" containerID="09fb7757e785afe73e877c523ea1d706aec5c7764fba9c6fd71e5d622123d9ff" exitCode=0
Oct 30 17:22:20.928857 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.928755    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64"
Oct 30 17:22:20.928857 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.928792    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64" event={"ID":"3f6eaf54-07a2-4010-9b79-e82670f15262","Type":"ContainerDied","Data":"09fb7757e785afe73e877c523ea1d706aec5c7764fba9c6fd71e5d622123d9ff"}
Oct 30 17:22:20.928857 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.928827    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64" event={"ID":"3f6eaf54-07a2-4010-9b79-e82670f15262","Type":"ContainerDied","Data":"d0aa33442297c1340c28dab02378c4e3456da05430d1c324e379b1f6daf4ade0"}
Oct 30 17:22:20.937508 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.937488    4433 scope.go:122] "RemoveContainer" containerID="f14b17f8f6814cc9e0771490a8163284320bf592c1137a85c50f5c565e6cf340"
Oct 30 17:22:20.937850 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:22:20.937833    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f14b17f8f6814cc9e0771490a8163284320bf592c1137a85c50f5c565e6cf340\": container with ID starting with f14b17f8f6814cc9e0771490a8163284320bf592c1137a85c50f5c565e6cf340 not found: ID does not exist" containerID="f14b17f8f6814cc9e0771490a8163284320bf592c1137a85c50f5c565e6cf340"
Oct 30 17:22:20.937930 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.937858    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f14b17f8f6814cc9e0771490a8163284320bf592c1137a85c50f5c565e6cf340"} err="failed to get container status \"f14b17f8f6814cc9e0771490a8163284320bf592c1137a85c50f5c565e6cf340\": rpc error: code = NotFound desc = could not find container \"f14b17f8f6814cc9e0771490a8163284320bf592c1137a85c50f5c565e6cf340\": container with ID starting with f14b17f8f6814cc9e0771490a8163284320bf592c1137a85c50f5c565e6cf340 not found: ID does not exist"
Oct 30 17:22:20.937930 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.937878    4433 scope.go:122] "RemoveContainer" containerID="09fb7757e785afe73e877c523ea1d706aec5c7764fba9c6fd71e5d622123d9ff"
Oct 30 17:22:20.947769 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.947752    4433 scope.go:122] "RemoveContainer" containerID="09fb7757e785afe73e877c523ea1d706aec5c7764fba9c6fd71e5d622123d9ff"
Oct 30 17:22:20.948085 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:22:20.948063    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09fb7757e785afe73e877c523ea1d706aec5c7764fba9c6fd71e5d622123d9ff\": container with ID starting with 09fb7757e785afe73e877c523ea1d706aec5c7764fba9c6fd71e5d622123d9ff not found: ID does not exist" containerID="09fb7757e785afe73e877c523ea1d706aec5c7764fba9c6fd71e5d622123d9ff"
Oct 30 17:22:20.948146 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.948093    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09fb7757e785afe73e877c523ea1d706aec5c7764fba9c6fd71e5d622123d9ff"} err="failed to get container status \"09fb7757e785afe73e877c523ea1d706aec5c7764fba9c6fd71e5d622123d9ff\": rpc error: code = NotFound desc = could not find container \"09fb7757e785afe73e877c523ea1d706aec5c7764fba9c6fd71e5d622123d9ff\": container with ID starting with 09fb7757e785afe73e877c523ea1d706aec5c7764fba9c6fd71e5d622123d9ff not found: ID does not exist"
Oct 30 17:22:20.992279 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.992243    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64"]
Oct 30 17:22:20.996961 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:20.996919    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-78c9c2/auth-test-model-78c9c2-kserve-79756ff877-vbs64"]
Oct 30 17:22:21.024683 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:21.024620    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2"]
Oct 30 17:22:21.028604 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:21.028555    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5d67ca-8ccc98f84-pjjd2"]
Oct 30 17:22:21.451477 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:21.450899    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq"]
Oct 30 17:22:21.708274 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:21.708190    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f6eaf54-07a2-4010-9b79-e82670f15262" path="/var/lib/kubelet/pods/3f6eaf54-07a2-4010-9b79-e82670f15262/volumes"
Oct 30 17:22:21.709820 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:21.709672    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb49dfad-1e85-4b96-9dd8-42374a708643" path="/var/lib/kubelet/pods/bb49dfad-1e85-4b96-9dd8-42374a708643/volumes"
Oct 30 17:22:21.710753 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:21.710730    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c07529e2-9bcd-4ec9-9c98-e0bb48c8effb" path="/var/lib/kubelet/pods/c07529e2-9bcd-4ec9-9c98-e0bb48c8effb/volumes"
Oct 30 17:22:21.712187 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:21.712074    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc12f255-74a5-4c88-b67b-a0c377797e84" path="/var/lib/kubelet/pods/fc12f255-74a5-4c88-b67b-a0c377797e84/volumes"
Oct 30 17:22:27.020427 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:22:27.019286    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:22:31.172993 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:31.172946    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:22:36.480257 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:36.480197    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq" podUID="dd03e357-fdbb-438b-847e-26ba74b88949" containerName="main" containerID="cri-o://4c52296e86cd72cef01ceb8425e710e1ff1334e1887ac74a6acd48ba75101df5" gracePeriod=45
Oct 30 17:22:36.769314 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:36.769289    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq"
Oct 30 17:22:36.836491 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:36.836456    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd03e357-fdbb-438b-847e-26ba74b88949-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd03e357-fdbb-438b-847e-26ba74b88949-dshm\") pod \"dd03e357-fdbb-438b-847e-26ba74b88949\" (UID: \"dd03e357-fdbb-438b-847e-26ba74b88949\") "
Oct 30 17:22:36.836669 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:36.836539    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd03e357-fdbb-438b-847e-26ba74b88949-kube-api-access-kdr2t\" (UniqueName: \"kubernetes.io/projected/dd03e357-fdbb-438b-847e-26ba74b88949-kube-api-access-kdr2t\") pod \"dd03e357-fdbb-438b-847e-26ba74b88949\" (UID: \"dd03e357-fdbb-438b-847e-26ba74b88949\") "
Oct 30 17:22:36.836669 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:36.836664    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd03e357-fdbb-438b-847e-26ba74b88949-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd03e357-fdbb-438b-847e-26ba74b88949-tmp-dir\") pod \"dd03e357-fdbb-438b-847e-26ba74b88949\" (UID: \"dd03e357-fdbb-438b-847e-26ba74b88949\") "
Oct 30 17:22:36.836756 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:36.836711    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd03e357-fdbb-438b-847e-26ba74b88949-home\" (UniqueName: \"kubernetes.io/empty-dir/dd03e357-fdbb-438b-847e-26ba74b88949-home\") pod \"dd03e357-fdbb-438b-847e-26ba74b88949\" (UID: \"dd03e357-fdbb-438b-847e-26ba74b88949\") "
Oct 30 17:22:36.836939 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:36.836916    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd03e357-fdbb-438b-847e-26ba74b88949-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd03e357-fdbb-438b-847e-26ba74b88949-model-cache\") pod \"dd03e357-fdbb-438b-847e-26ba74b88949\" (UID: \"dd03e357-fdbb-438b-847e-26ba74b88949\") "
Oct 30 17:22:36.837035 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:36.836962    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd03e357-fdbb-438b-847e-26ba74b88949-tls-certs\" (UniqueName: \"kubernetes.io/secret/dd03e357-fdbb-438b-847e-26ba74b88949-tls-certs\") pod \"dd03e357-fdbb-438b-847e-26ba74b88949\" (UID: \"dd03e357-fdbb-438b-847e-26ba74b88949\") "
Oct 30 17:22:36.837124 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:36.837081    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd03e357-fdbb-438b-847e-26ba74b88949-tmp-dir" pod "dd03e357-fdbb-438b-847e-26ba74b88949" (UID: "dd03e357-fdbb-438b-847e-26ba74b88949"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:22:36.837290 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:36.837253    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd03e357-fdbb-438b-847e-26ba74b88949-model-cache" pod "dd03e357-fdbb-438b-847e-26ba74b88949" (UID: "dd03e357-fdbb-438b-847e-26ba74b88949"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:22:36.837508 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:36.837484    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd03e357-fdbb-438b-847e-26ba74b88949-home" pod "dd03e357-fdbb-438b-847e-26ba74b88949" (UID: "dd03e357-fdbb-438b-847e-26ba74b88949"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:22:36.837996 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:36.837968    4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd03e357-fdbb-438b-847e-26ba74b88949-tmp-dir\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:36.837996 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:36.837999    4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd03e357-fdbb-438b-847e-26ba74b88949-home\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:36.838159 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:36.838014    4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd03e357-fdbb-438b-847e-26ba74b88949-model-cache\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:36.839136 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:36.839110    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd03e357-fdbb-438b-847e-26ba74b88949-dshm" pod "dd03e357-fdbb-438b-847e-26ba74b88949" (UID: "dd03e357-fdbb-438b-847e-26ba74b88949"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:22:36.839384 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:36.839356    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd03e357-fdbb-438b-847e-26ba74b88949-kube-api-access-kdr2t" pod "dd03e357-fdbb-438b-847e-26ba74b88949" (UID: "dd03e357-fdbb-438b-847e-26ba74b88949"). InnerVolumeSpecName "kube-api-access-kdr2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:22:36.839627 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:36.839603    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd03e357-fdbb-438b-847e-26ba74b88949-tls-certs" pod "dd03e357-fdbb-438b-847e-26ba74b88949" (UID: "dd03e357-fdbb-438b-847e-26ba74b88949"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:22:36.939186 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:36.939155    4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd03e357-fdbb-438b-847e-26ba74b88949-tls-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:36.939186 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:36.939183    4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd03e357-fdbb-438b-847e-26ba74b88949-dshm\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:36.939186 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:36.939193    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdr2t\" (UniqueName: \"kubernetes.io/projected/dd03e357-fdbb-438b-847e-26ba74b88949-kube-api-access-kdr2t\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:22:37.056187 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:37.056098    4433 generic.go:356] "Generic (PLEG): container finished" podID="dd03e357-fdbb-438b-847e-26ba74b88949" containerID="4c52296e86cd72cef01ceb8425e710e1ff1334e1887ac74a6acd48ba75101df5" exitCode=0
Oct 30 17:22:37.056187 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:37.056157    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq"
Oct 30 17:22:37.056369 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:37.056306    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq" event={"ID":"dd03e357-fdbb-438b-847e-26ba74b88949","Type":"ContainerDied","Data":"4c52296e86cd72cef01ceb8425e710e1ff1334e1887ac74a6acd48ba75101df5"}
Oct 30 17:22:37.056369 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:37.056340    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq" event={"ID":"dd03e357-fdbb-438b-847e-26ba74b88949","Type":"ContainerDied","Data":"2f327d657aacf3920a082880aefa27009b4123088c519b1e54d3642e6e86ac07"}
Oct 30 17:22:37.056369 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:37.056361    4433 scope.go:122] "RemoveContainer" containerID="4c52296e86cd72cef01ceb8425e710e1ff1334e1887ac74a6acd48ba75101df5"
Oct 30 17:22:37.067784 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:37.067765    4433 scope.go:122] "RemoveContainer" containerID="4c52296e86cd72cef01ceb8425e710e1ff1334e1887ac74a6acd48ba75101df5"
Oct 30 17:22:37.068098 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:22:37.068077    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c52296e86cd72cef01ceb8425e710e1ff1334e1887ac74a6acd48ba75101df5\": container with ID starting with 4c52296e86cd72cef01ceb8425e710e1ff1334e1887ac74a6acd48ba75101df5 not found: ID does not exist" containerID="4c52296e86cd72cef01ceb8425e710e1ff1334e1887ac74a6acd48ba75101df5"
Oct 30 17:22:37.068164 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:37.068109    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c52296e86cd72cef01ceb8425e710e1ff1334e1887ac74a6acd48ba75101df5"} err="failed to get container status \"4c52296e86cd72cef01ceb8425e710e1ff1334e1887ac74a6acd48ba75101df5\": rpc error: code = NotFound desc = could not find container \"4c52296e86cd72cef01ceb8425e710e1ff1334e1887ac74a6acd48ba75101df5\": container with ID starting with 4c52296e86cd72cef01ceb8425e710e1ff1334e1887ac74a6acd48ba75101df5 not found: ID does not exist"
Oct 30 17:22:37.109484 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:37.109457    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq"]
Oct 30 17:22:37.112054 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:37.112032    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-7c52f8/test-model-7c52f8-kserve-5c9f8d47df-rsxwq"]
Oct 30 17:22:37.708613 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:37.708575    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd03e357-fdbb-438b-847e-26ba74b88949" path="/var/lib/kubelet/pods/dd03e357-fdbb-438b-847e-26ba74b88949/volumes"
Oct 30 17:22:53.545194 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:22:53.544138    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86"]
Oct 30 17:22:56.978706 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:22:56.978664    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:23:08.583279 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:08.583221    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86" podUID="5420c28a-0d07-47df-8d12-1ec97ec9707e" containerName="main" containerID="cri-o://b311e5f9c34e6765bde2101bacc9817e84ec0dc1828ea1f7a61d0d88cea4c3dc" gracePeriod=45
Oct 30 17:23:08.859383 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:08.859351    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86"
Oct 30 17:23:09.050874 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:09.050838    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5420c28a-0d07-47df-8d12-1ec97ec9707e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5420c28a-0d07-47df-8d12-1ec97ec9707e-tmp-dir\") pod \"5420c28a-0d07-47df-8d12-1ec97ec9707e\" (UID: \"5420c28a-0d07-47df-8d12-1ec97ec9707e\") "
Oct 30 17:23:09.051076 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:09.050954    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5420c28a-0d07-47df-8d12-1ec97ec9707e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5420c28a-0d07-47df-8d12-1ec97ec9707e-model-cache\") pod \"5420c28a-0d07-47df-8d12-1ec97ec9707e\" (UID: \"5420c28a-0d07-47df-8d12-1ec97ec9707e\") "
Oct 30 17:23:09.051076 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:09.051004    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5420c28a-0d07-47df-8d12-1ec97ec9707e-tls-certs\" (UniqueName: \"kubernetes.io/secret/5420c28a-0d07-47df-8d12-1ec97ec9707e-tls-certs\") pod \"5420c28a-0d07-47df-8d12-1ec97ec9707e\" (UID: \"5420c28a-0d07-47df-8d12-1ec97ec9707e\") "
Oct 30 17:23:09.051076 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:09.051033    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5420c28a-0d07-47df-8d12-1ec97ec9707e-home\" (UniqueName: \"kubernetes.io/empty-dir/5420c28a-0d07-47df-8d12-1ec97ec9707e-home\") pod \"5420c28a-0d07-47df-8d12-1ec97ec9707e\" (UID: \"5420c28a-0d07-47df-8d12-1ec97ec9707e\") "
Oct 30 17:23:09.051076 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:09.051068    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5420c28a-0d07-47df-8d12-1ec97ec9707e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5420c28a-0d07-47df-8d12-1ec97ec9707e-dshm\") pod \"5420c28a-0d07-47df-8d12-1ec97ec9707e\" (UID: \"5420c28a-0d07-47df-8d12-1ec97ec9707e\") "
Oct 30 17:23:09.051288 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:09.051156    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5420c28a-0d07-47df-8d12-1ec97ec9707e-kube-api-access-xrvgf\" (UniqueName: \"kubernetes.io/projected/5420c28a-0d07-47df-8d12-1ec97ec9707e-kube-api-access-xrvgf\") pod \"5420c28a-0d07-47df-8d12-1ec97ec9707e\" (UID: \"5420c28a-0d07-47df-8d12-1ec97ec9707e\") "
Oct 30 17:23:09.051288 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:09.051166    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5420c28a-0d07-47df-8d12-1ec97ec9707e-tmp-dir" pod "5420c28a-0d07-47df-8d12-1ec97ec9707e" (UID: "5420c28a-0d07-47df-8d12-1ec97ec9707e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:23:09.051703 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:09.051677    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5420c28a-0d07-47df-8d12-1ec97ec9707e-model-cache" pod "5420c28a-0d07-47df-8d12-1ec97ec9707e" (UID: "5420c28a-0d07-47df-8d12-1ec97ec9707e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:23:09.051883 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:09.051709    4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5420c28a-0d07-47df-8d12-1ec97ec9707e-tmp-dir\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:23:09.052099 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:09.052076    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5420c28a-0d07-47df-8d12-1ec97ec9707e-home" pod "5420c28a-0d07-47df-8d12-1ec97ec9707e" (UID: "5420c28a-0d07-47df-8d12-1ec97ec9707e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:23:09.053806 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:09.053780    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5420c28a-0d07-47df-8d12-1ec97ec9707e-tls-certs" pod "5420c28a-0d07-47df-8d12-1ec97ec9707e" (UID: "5420c28a-0d07-47df-8d12-1ec97ec9707e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:23:09.054049 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:09.054026    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5420c28a-0d07-47df-8d12-1ec97ec9707e-kube-api-access-xrvgf" pod "5420c28a-0d07-47df-8d12-1ec97ec9707e" (UID: "5420c28a-0d07-47df-8d12-1ec97ec9707e"). InnerVolumeSpecName "kube-api-access-xrvgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:23:09.063010 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:09.062988    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5420c28a-0d07-47df-8d12-1ec97ec9707e-dshm" pod "5420c28a-0d07-47df-8d12-1ec97ec9707e" (UID: "5420c28a-0d07-47df-8d12-1ec97ec9707e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:23:09.153363 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:09.153267    4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5420c28a-0d07-47df-8d12-1ec97ec9707e-model-cache\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:23:09.153363 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:09.153306    4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5420c28a-0d07-47df-8d12-1ec97ec9707e-tls-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:23:09.153363 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:09.153318    4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5420c28a-0d07-47df-8d12-1ec97ec9707e-home\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:23:09.153363 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:09.153328    4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5420c28a-0d07-47df-8d12-1ec97ec9707e-dshm\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:23:09.153363 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:09.153340    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrvgf\" (UniqueName: \"kubernetes.io/projected/5420c28a-0d07-47df-8d12-1ec97ec9707e-kube-api-access-xrvgf\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:23:09.277986 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:09.277946    4433 generic.go:356] "Generic (PLEG): container finished" podID="5420c28a-0d07-47df-8d12-1ec97ec9707e" containerID="b311e5f9c34e6765bde2101bacc9817e84ec0dc1828ea1f7a61d0d88cea4c3dc" exitCode=0
Oct 30 17:23:09.278189 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:09.278006    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86"
Oct 30 17:23:09.278189 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:09.278027    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86" event={"ID":"5420c28a-0d07-47df-8d12-1ec97ec9707e","Type":"ContainerDied","Data":"b311e5f9c34e6765bde2101bacc9817e84ec0dc1828ea1f7a61d0d88cea4c3dc"}
Oct 30 17:23:09.278189 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:09.278065    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86" event={"ID":"5420c28a-0d07-47df-8d12-1ec97ec9707e","Type":"ContainerDied","Data":"50e8c74907c24b2f477d85b057036de768cf4de0331ba962da89c70a01f97886"}
Oct 30 17:23:09.278189 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:09.278086    4433 scope.go:122] "RemoveContainer" containerID="b311e5f9c34e6765bde2101bacc9817e84ec0dc1828ea1f7a61d0d88cea4c3dc"
Oct 30 17:23:09.289131 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:09.289093    4433 scope.go:122] "RemoveContainer" containerID="b311e5f9c34e6765bde2101bacc9817e84ec0dc1828ea1f7a61d0d88cea4c3dc"
Oct 30 17:23:09.289395 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:23:09.289373    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b311e5f9c34e6765bde2101bacc9817e84ec0dc1828ea1f7a61d0d88cea4c3dc\": container with ID starting with b311e5f9c34e6765bde2101bacc9817e84ec0dc1828ea1f7a61d0d88cea4c3dc not found: ID does not exist" containerID="b311e5f9c34e6765bde2101bacc9817e84ec0dc1828ea1f7a61d0d88cea4c3dc"
Oct 30 17:23:09.289591 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:09.289423    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b311e5f9c34e6765bde2101bacc9817e84ec0dc1828ea1f7a61d0d88cea4c3dc"} err="failed to get container status \"b311e5f9c34e6765bde2101bacc9817e84ec0dc1828ea1f7a61d0d88cea4c3dc\": rpc error: code = NotFound desc = could not find container \"b311e5f9c34e6765bde2101bacc9817e84ec0dc1828ea1f7a61d0d88cea4c3dc\": container with ID starting with b311e5f9c34e6765bde2101bacc9817e84ec0dc1828ea1f7a61d0d88cea4c3dc not found: ID does not exist"
Oct 30 17:23:09.328237 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:09.328206    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86"]
Oct 30 17:23:09.339561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:09.339526    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-dd5c8f/test-model-dd5c8f-kserve-554d64c6dc-6vm86"]
Oct 30 17:23:09.695843 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:09.695808    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5420c28a-0d07-47df-8d12-1ec97ec9707e" path="/var/lib/kubelet/pods/5420c28a-0d07-47df-8d12-1ec97ec9707e/volumes"
Oct 30 17:23:26.898358 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:26.898323    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb"]
Oct 30 17:23:26.898967 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:26.898594    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb" podUID="eb27ba4a-4385-4f07-a202-f8f3bccb6559" containerName="maas-api" containerID="cri-o://faf2df524894edf5df4fbde63fede5d1594b671ed691324b4e59faaec7c57709" gracePeriod=30
Oct 30 17:23:26.993993 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:23:26.993949    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:23:27.533427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:27.527984    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb"
Oct 30 17:23:27.533427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:27.528516    4433 generic.go:356] "Generic (PLEG): container finished" podID="eb27ba4a-4385-4f07-a202-f8f3bccb6559" containerID="faf2df524894edf5df4fbde63fede5d1594b671ed691324b4e59faaec7c57709" exitCode=0
Oct 30 17:23:27.533427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:27.528672    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb" event={"ID":"eb27ba4a-4385-4f07-a202-f8f3bccb6559","Type":"ContainerDied","Data":"faf2df524894edf5df4fbde63fede5d1594b671ed691324b4e59faaec7c57709"}
Oct 30 17:23:27.533427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:27.528730    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb" event={"ID":"eb27ba4a-4385-4f07-a202-f8f3bccb6559","Type":"ContainerDied","Data":"30b5d4ec005e55058dd6edbd3f9daf07ff80c93f9879fca8eab758446205eac5"}
Oct 30 17:23:27.533427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:27.528770    4433 scope.go:122] "RemoveContainer" containerID="faf2df524894edf5df4fbde63fede5d1594b671ed691324b4e59faaec7c57709"
Oct 30 17:23:27.655591 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:27.653987    4433 scope.go:122] "RemoveContainer" containerID="faf2df524894edf5df4fbde63fede5d1594b671ed691324b4e59faaec7c57709"
Oct 30 17:23:27.656058 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:23:27.655933    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faf2df524894edf5df4fbde63fede5d1594b671ed691324b4e59faaec7c57709\": container with ID starting with faf2df524894edf5df4fbde63fede5d1594b671ed691324b4e59faaec7c57709 not found: ID does not exist" containerID="faf2df524894edf5df4fbde63fede5d1594b671ed691324b4e59faaec7c57709"
Oct 30 17:23:27.656058 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:27.655975    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faf2df524894edf5df4fbde63fede5d1594b671ed691324b4e59faaec7c57709"} err="failed to get container status \"faf2df524894edf5df4fbde63fede5d1594b671ed691324b4e59faaec7c57709\": rpc error: code = NotFound desc = could not find container \"faf2df524894edf5df4fbde63fede5d1594b671ed691324b4e59faaec7c57709\": container with ID starting with faf2df524894edf5df4fbde63fede5d1594b671ed691324b4e59faaec7c57709 not found: ID does not exist"
Oct 30 17:23:27.716085 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:27.714334    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb27ba4a-4385-4f07-a202-f8f3bccb6559-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb27ba4a-4385-4f07-a202-f8f3bccb6559-metrics-certs\") pod \"eb27ba4a-4385-4f07-a202-f8f3bccb6559\" (UID: \"eb27ba4a-4385-4f07-a202-f8f3bccb6559\") "
Oct 30 17:23:27.716085 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:27.714459    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb27ba4a-4385-4f07-a202-f8f3bccb6559-kube-api-access-2vmdx\" (UniqueName: \"kubernetes.io/projected/eb27ba4a-4385-4f07-a202-f8f3bccb6559-kube-api-access-2vmdx\") pod \"eb27ba4a-4385-4f07-a202-f8f3bccb6559\" (UID: \"eb27ba4a-4385-4f07-a202-f8f3bccb6559\") "
Oct 30 17:23:27.716085 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:27.715048    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb27ba4a-4385-4f07-a202-f8f3bccb6559-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb27ba4a-4385-4f07-a202-f8f3bccb6559-openshift-service-ca\") pod \"eb27ba4a-4385-4f07-a202-f8f3bccb6559\" (UID: \"eb27ba4a-4385-4f07-a202-f8f3bccb6559\") "
Oct 30 17:23:27.716085 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:27.715147    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb27ba4a-4385-4f07-a202-f8f3bccb6559-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb27ba4a-4385-4f07-a202-f8f3bccb6559-maas-api-tls\") pod \"eb27ba4a-4385-4f07-a202-f8f3bccb6559\" (UID: \"eb27ba4a-4385-4f07-a202-f8f3bccb6559\") "
Oct 30 17:23:27.716085 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:27.715599    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb27ba4a-4385-4f07-a202-f8f3bccb6559-openshift-service-ca" pod "eb27ba4a-4385-4f07-a202-f8f3bccb6559" (UID: "eb27ba4a-4385-4f07-a202-f8f3bccb6559"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:23:27.717212 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:27.717183    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb27ba4a-4385-4f07-a202-f8f3bccb6559-metrics-certs" pod "eb27ba4a-4385-4f07-a202-f8f3bccb6559" (UID: "eb27ba4a-4385-4f07-a202-f8f3bccb6559"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:23:27.717894 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:27.717868    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb27ba4a-4385-4f07-a202-f8f3bccb6559-kube-api-access-2vmdx" pod "eb27ba4a-4385-4f07-a202-f8f3bccb6559" (UID: "eb27ba4a-4385-4f07-a202-f8f3bccb6559"). InnerVolumeSpecName "kube-api-access-2vmdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:23:27.718310 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:27.718289    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb27ba4a-4385-4f07-a202-f8f3bccb6559-maas-api-tls" pod "eb27ba4a-4385-4f07-a202-f8f3bccb6559" (UID: "eb27ba4a-4385-4f07-a202-f8f3bccb6559"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:23:27.816577 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:27.816546    4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb27ba4a-4385-4f07-a202-f8f3bccb6559-openshift-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:23:27.816577 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:27.816574    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb27ba4a-4385-4f07-a202-f8f3bccb6559-maas-api-tls\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:23:27.816577 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:27.816583    4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb27ba4a-4385-4f07-a202-f8f3bccb6559-metrics-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:23:27.816820 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:27.816591    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vmdx\" (UniqueName: \"kubernetes.io/projected/eb27ba4a-4385-4f07-a202-f8f3bccb6559-kube-api-access-2vmdx\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:23:27.932656 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:27.932616    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:23:28.535835 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:28.535803    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb"
Oct 30 17:23:28.618084 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:28.618045    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb"]
Oct 30 17:23:28.629455 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:28.629421    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7c52f8-7bcf84965d-g9bsb"]
Oct 30 17:23:29.995973 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:29.995929    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb27ba4a-4385-4f07-a202-f8f3bccb6559" path="/var/lib/kubelet/pods/eb27ba4a-4385-4f07-a202-f8f3bccb6559/volumes"
Oct 30 17:23:35.256040 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:35.256001    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"]
Oct 30 17:23:35.256555 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:35.256272    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm" podUID="f265af5b-6027-4101-99db-328a61fd85da" containerName="istio-proxy" containerID="cri-o://524e209f4c21b9a4f87668ad8294bb6500f9b55a7fcc506c739bbada9d73c4fd" gracePeriod=30
Oct 30 17:23:35.272967 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:35.272932    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"]
Oct 30 17:23:35.273197 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:35.273172    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv" podUID="103bc667-ffe4-4cc3-a66d-fc85ed8a0a51" containerName="istio-proxy" containerID="cri-o://a913ca1e0159a6ef5bd341ff843691ed2efb9e8ca38e53cdb110e9d4eafb3a9c" gracePeriod=30
Oct 30 17:23:36.612474 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:36.610510    4433 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:23:36.612474 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:36.610580    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm" podUID="f265af5b-6027-4101-99db-328a61fd85da" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:23:38.135502 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:38.135453    4433 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:23:38.135934 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:38.135538    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv" podUID="103bc667-ffe4-4cc3-a66d-fc85ed8a0a51" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:23:40.552696 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.552663    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"
Oct 30 17:23:40.593159 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.593132    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"
Oct 30 17:23:40.636253 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.636218    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f265af5b-6027-4101-99db-328a61fd85da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f265af5b-6027-4101-99db-328a61fd85da-workload-certs\") pod \"f265af5b-6027-4101-99db-328a61fd85da\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") "
Oct 30 17:23:40.636439 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.636266    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f265af5b-6027-4101-99db-328a61fd85da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f265af5b-6027-4101-99db-328a61fd85da-istio-data\") pod \"f265af5b-6027-4101-99db-328a61fd85da\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") "
Oct 30 17:23:40.636439 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.636330    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f265af5b-6027-4101-99db-328a61fd85da-istio-token\" (UniqueName: \"kubernetes.io/projected/f265af5b-6027-4101-99db-328a61fd85da-istio-token\") pod \"f265af5b-6027-4101-99db-328a61fd85da\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") "
Oct 30 17:23:40.636439 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.636378    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f265af5b-6027-4101-99db-328a61fd85da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f265af5b-6027-4101-99db-328a61fd85da-istio-envoy\") pod \"f265af5b-6027-4101-99db-328a61fd85da\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") "
Oct 30 17:23:40.636566 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.636479    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f265af5b-6027-4101-99db-328a61fd85da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f265af5b-6027-4101-99db-328a61fd85da-istio-podinfo\") pod \"f265af5b-6027-4101-99db-328a61fd85da\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") "
Oct 30 17:23:40.636566 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.636514    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f265af5b-6027-4101-99db-328a61fd85da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f265af5b-6027-4101-99db-328a61fd85da-istiod-ca-cert\") pod \"f265af5b-6027-4101-99db-328a61fd85da\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") "
Oct 30 17:23:40.636566 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.636551    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f265af5b-6027-4101-99db-328a61fd85da-workload-certs" pod "f265af5b-6027-4101-99db-328a61fd85da" (UID: "f265af5b-6027-4101-99db-328a61fd85da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:23:40.636666 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.636588    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f265af5b-6027-4101-99db-328a61fd85da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f265af5b-6027-4101-99db-328a61fd85da-credential-socket\") pod \"f265af5b-6027-4101-99db-328a61fd85da\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") "
Oct 30 17:23:40.636666 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.636633    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f265af5b-6027-4101-99db-328a61fd85da-istio-data" pod "f265af5b-6027-4101-99db-328a61fd85da" (UID: "f265af5b-6027-4101-99db-328a61fd85da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:23:40.636666 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.636655    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f265af5b-6027-4101-99db-328a61fd85da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f265af5b-6027-4101-99db-328a61fd85da-workload-socket\") pod \"f265af5b-6027-4101-99db-328a61fd85da\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") "
Oct 30 17:23:40.637109 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.637010    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f265af5b-6027-4101-99db-328a61fd85da-kube-api-access-q57kp\" (UniqueName: \"kubernetes.io/projected/f265af5b-6027-4101-99db-328a61fd85da-kube-api-access-q57kp\") pod \"f265af5b-6027-4101-99db-328a61fd85da\" (UID: \"f265af5b-6027-4101-99db-328a61fd85da\") "
Oct 30 17:23:40.637109 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.637065    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f265af5b-6027-4101-99db-328a61fd85da-credential-socket" pod "f265af5b-6027-4101-99db-328a61fd85da" (UID: "f265af5b-6027-4101-99db-328a61fd85da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:23:40.637319 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.637158    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f265af5b-6027-4101-99db-328a61fd85da-istiod-ca-cert" pod "f265af5b-6027-4101-99db-328a61fd85da" (UID: "f265af5b-6027-4101-99db-328a61fd85da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:23:40.637319 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.637224    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f265af5b-6027-4101-99db-328a61fd85da-workload-socket" pod "f265af5b-6027-4101-99db-328a61fd85da" (UID: "f265af5b-6027-4101-99db-328a61fd85da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:23:40.637679 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.637655    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f265af5b-6027-4101-99db-328a61fd85da-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:23:40.637779 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.637685    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f265af5b-6027-4101-99db-328a61fd85da-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:23:40.637779 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.637698    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f265af5b-6027-4101-99db-328a61fd85da-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:23:40.637779 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.637711    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f265af5b-6027-4101-99db-328a61fd85da-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:23:40.637779 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.637723    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f265af5b-6027-4101-99db-328a61fd85da-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:23:40.638913 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.638889    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f265af5b-6027-4101-99db-328a61fd85da-istio-envoy" pod "f265af5b-6027-4101-99db-328a61fd85da" (UID: "f265af5b-6027-4101-99db-328a61fd85da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:23:40.639221 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.639196    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f265af5b-6027-4101-99db-328a61fd85da-istio-token" pod "f265af5b-6027-4101-99db-328a61fd85da" (UID: "f265af5b-6027-4101-99db-328a61fd85da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:23:40.639328 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.639273    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f265af5b-6027-4101-99db-328a61fd85da-istio-podinfo" pod "f265af5b-6027-4101-99db-328a61fd85da" (UID: "f265af5b-6027-4101-99db-328a61fd85da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:23:40.639820 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.639797    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f265af5b-6027-4101-99db-328a61fd85da-kube-api-access-q57kp" pod "f265af5b-6027-4101-99db-328a61fd85da" (UID: "f265af5b-6027-4101-99db-328a61fd85da"). InnerVolumeSpecName "kube-api-access-q57kp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:23:40.739062 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.739027    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-workload-certs\") pod \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") "
Oct 30 17:23:40.739233 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.739097    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-kube-api-access-ffx8d\" (UniqueName: \"kubernetes.io/projected/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-kube-api-access-ffx8d\") pod \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") "
Oct 30 17:23:40.739233 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.739177    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-workload-socket\") pod \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") "
Oct 30 17:23:40.739369 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.739268    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-istio-token\" (UniqueName: \"kubernetes.io/projected/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-istio-token\") pod \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") "
Oct 30 17:23:40.739369 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.739326    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-istiod-ca-cert\") pod \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") "
Oct 30 17:23:40.739369 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.739359    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-istio-data\") pod \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") "
Oct 30 17:23:40.739589 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.739504    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-istio-envoy\") pod \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") "
Oct 30 17:23:40.739589 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.739559    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-istio-podinfo\") pod \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") "
Oct 30 17:23:40.739682 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.739590    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-credential-socket\") pod \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\" (UID: \"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51\") "
Oct 30 17:23:40.739682 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.739640    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-workload-socket" pod "103bc667-ffe4-4cc3-a66d-fc85ed8a0a51" (UID: "103bc667-ffe4-4cc3-a66d-fc85ed8a0a51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:23:40.740096 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.740055    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-workload-certs" pod "103bc667-ffe4-4cc3-a66d-fc85ed8a0a51" (UID: "103bc667-ffe4-4cc3-a66d-fc85ed8a0a51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:23:40.740096 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.740064    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-istio-data" pod "103bc667-ffe4-4cc3-a66d-fc85ed8a0a51" (UID: "103bc667-ffe4-4cc3-a66d-fc85ed8a0a51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:23:40.740306 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.740277    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-credential-socket" pod "103bc667-ffe4-4cc3-a66d-fc85ed8a0a51" (UID: "103bc667-ffe4-4cc3-a66d-fc85ed8a0a51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:23:40.740392 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.740368    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q57kp\" (UniqueName: \"kubernetes.io/projected/f265af5b-6027-4101-99db-328a61fd85da-kube-api-access-q57kp\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:23:40.740503 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.740398    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:23:40.740503 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.740429    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:23:40.740503 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.740443    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f265af5b-6027-4101-99db-328a61fd85da-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:23:40.740503 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.740455    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f265af5b-6027-4101-99db-328a61fd85da-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:23:40.740503 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.740466    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f265af5b-6027-4101-99db-328a61fd85da-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:23:40.740503 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.740478    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:23:40.740997 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.740973    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-istiod-ca-cert" pod "103bc667-ffe4-4cc3-a66d-fc85ed8a0a51" (UID: "103bc667-ffe4-4cc3-a66d-fc85ed8a0a51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:23:40.742823 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.742793    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-istio-envoy" pod "103bc667-ffe4-4cc3-a66d-fc85ed8a0a51" (UID: "103bc667-ffe4-4cc3-a66d-fc85ed8a0a51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:23:40.742823 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.742795    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-kube-api-access-ffx8d" pod "103bc667-ffe4-4cc3-a66d-fc85ed8a0a51" (UID: "103bc667-ffe4-4cc3-a66d-fc85ed8a0a51"). InnerVolumeSpecName "kube-api-access-ffx8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:23:40.743010 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.742849    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-istio-podinfo" pod "103bc667-ffe4-4cc3-a66d-fc85ed8a0a51" (UID: "103bc667-ffe4-4cc3-a66d-fc85ed8a0a51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:23:40.743226 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.743210    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-istio-token" pod "103bc667-ffe4-4cc3-a66d-fc85ed8a0a51" (UID: "103bc667-ffe4-4cc3-a66d-fc85ed8a0a51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:23:40.841800 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.841718    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffx8d\" (UniqueName: \"kubernetes.io/projected/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-kube-api-access-ffx8d\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:23:40.841800 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.841747    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:23:40.841800 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.841762    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:23:40.841800 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.841772    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:23:40.841800 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.841781    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:23:40.841800 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.841789    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:23:40.931496 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.931454    4433 generic.go:356] "Generic (PLEG): container finished" podID="f265af5b-6027-4101-99db-328a61fd85da" containerID="524e209f4c21b9a4f87668ad8294bb6500f9b55a7fcc506c739bbada9d73c4fd" exitCode=0
Oct 30 17:23:40.931687 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.931512    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"
Oct 30 17:23:40.931687 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.931560    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm" event={"ID":"f265af5b-6027-4101-99db-328a61fd85da","Type":"ContainerDied","Data":"524e209f4c21b9a4f87668ad8294bb6500f9b55a7fcc506c739bbada9d73c4fd"}
Oct 30 17:23:40.931687 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.931604    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm" event={"ID":"f265af5b-6027-4101-99db-328a61fd85da","Type":"ContainerDied","Data":"1217aad569e894409c0885012fce40ea9894e7e8efba1ad96fdea9b68585c700"}
Oct 30 17:23:40.931687 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.931633    4433 scope.go:122] "RemoveContainer" containerID="524e209f4c21b9a4f87668ad8294bb6500f9b55a7fcc506c739bbada9d73c4fd"
Oct 30 17:23:40.932993 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.932966    4433 generic.go:356] "Generic (PLEG): container finished" podID="103bc667-ffe4-4cc3-a66d-fc85ed8a0a51" containerID="a913ca1e0159a6ef5bd341ff843691ed2efb9e8ca38e53cdb110e9d4eafb3a9c" exitCode=0
Oct 30 17:23:40.933096 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.933023    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"
Oct 30 17:23:40.933161 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.933024    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv" event={"ID":"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51","Type":"ContainerDied","Data":"a913ca1e0159a6ef5bd341ff843691ed2efb9e8ca38e53cdb110e9d4eafb3a9c"}
Oct 30 17:23:40.933220 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.933175    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv" event={"ID":"103bc667-ffe4-4cc3-a66d-fc85ed8a0a51","Type":"ContainerDied","Data":"c1fa43edc92f8850c9d9f3c46fd16345ce64beb7851b79c0a500f71623094283"}
Oct 30 17:23:40.943915 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.943875    4433 scope.go:122] "RemoveContainer" containerID="524e209f4c21b9a4f87668ad8294bb6500f9b55a7fcc506c739bbada9d73c4fd"
Oct 30 17:23:40.944311 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:23:40.944290    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"524e209f4c21b9a4f87668ad8294bb6500f9b55a7fcc506c739bbada9d73c4fd\": container with ID starting with 524e209f4c21b9a4f87668ad8294bb6500f9b55a7fcc506c739bbada9d73c4fd not found: ID does not exist" containerID="524e209f4c21b9a4f87668ad8294bb6500f9b55a7fcc506c739bbada9d73c4fd"
Oct 30 17:23:40.944420 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.944319    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"524e209f4c21b9a4f87668ad8294bb6500f9b55a7fcc506c739bbada9d73c4fd"} err="failed to get container status \"524e209f4c21b9a4f87668ad8294bb6500f9b55a7fcc506c739bbada9d73c4fd\": rpc error: code = NotFound desc = could not find container \"524e209f4c21b9a4f87668ad8294bb6500f9b55a7fcc506c739bbada9d73c4fd\": container with ID starting with 524e209f4c21b9a4f87668ad8294bb6500f9b55a7fcc506c739bbada9d73c4fd not found: ID does not exist"
Oct 30 17:23:40.944420 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.944345    4433 scope.go:122] "RemoveContainer" containerID="a913ca1e0159a6ef5bd341ff843691ed2efb9e8ca38e53cdb110e9d4eafb3a9c"
Oct 30 17:23:40.954208 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.954190    4433 scope.go:122] "RemoveContainer" containerID="a913ca1e0159a6ef5bd341ff843691ed2efb9e8ca38e53cdb110e9d4eafb3a9c"
Oct 30 17:23:40.954479 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:23:40.954456    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a913ca1e0159a6ef5bd341ff843691ed2efb9e8ca38e53cdb110e9d4eafb3a9c\": container with ID starting with a913ca1e0159a6ef5bd341ff843691ed2efb9e8ca38e53cdb110e9d4eafb3a9c not found: ID does not exist" containerID="a913ca1e0159a6ef5bd341ff843691ed2efb9e8ca38e53cdb110e9d4eafb3a9c"
Oct 30 17:23:40.954546 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.954486    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a913ca1e0159a6ef5bd341ff843691ed2efb9e8ca38e53cdb110e9d4eafb3a9c"} err="failed to get container status \"a913ca1e0159a6ef5bd341ff843691ed2efb9e8ca38e53cdb110e9d4eafb3a9c\": rpc error: code = NotFound desc = could not find container \"a913ca1e0159a6ef5bd341ff843691ed2efb9e8ca38e53cdb110e9d4eafb3a9c\": container with ID starting with a913ca1e0159a6ef5bd341ff843691ed2efb9e8ca38e53cdb110e9d4eafb3a9c not found: ID does not exist"
Oct 30 17:23:40.984583 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.984551    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"]
Oct 30 17:23:40.996023 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:40.995993    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-kfvfm"]
Oct 30 17:23:41.023022 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:41.022991    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"]
Oct 30 17:23:41.027335 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:41.027305    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-zlbdv"]
Oct 30 17:23:41.696344 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:41.696301    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="103bc667-ffe4-4cc3-a66d-fc85ed8a0a51" path="/var/lib/kubelet/pods/103bc667-ffe4-4cc3-a66d-fc85ed8a0a51/volumes"
Oct 30 17:23:41.697037 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:41.697015    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f265af5b-6027-4101-99db-328a61fd85da" path="/var/lib/kubelet/pods/f265af5b-6027-4101-99db-328a61fd85da/volumes"
Oct 30 17:23:52.915024 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:52.914947    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"]
Oct 30 17:23:52.915670 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:52.915638    4433 cpu_manager.go:414] "containerMap: removing container" podUID="103bc667-ffe4-4cc3-a66d-fc85ed8a0a51" containerName="istio-proxy"
Oct 30 17:23:52.915670 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:52.915664    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="103bc667-ffe4-4cc3-a66d-fc85ed8a0a51" containerName="istio-proxy"
Oct 30 17:23:52.915874 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:52.915679    4433 cpu_manager.go:414] "containerMap: removing container" podUID="bb49dfad-1e85-4b96-9dd8-42374a708643" containerName="istio-proxy"
Oct 30 17:23:52.915874 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:52.915688    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb49dfad-1e85-4b96-9dd8-42374a708643" containerName="istio-proxy"
Oct 30 17:23:52.915874 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:52.915703    4433 cpu_manager.go:414] "containerMap: removing container" podUID="dd03e357-fdbb-438b-847e-26ba74b88949" containerName="main"
Oct 30 17:23:52.915874 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:52.915710    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd03e357-fdbb-438b-847e-26ba74b88949" containerName="main"
Oct 30 17:23:52.915874 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:52.915734    4433 cpu_manager.go:414] "containerMap: removing container" podUID="5420c28a-0d07-47df-8d12-1ec97ec9707e" containerName="main"
Oct 30 17:23:52.915874 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:52.915742    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5420c28a-0d07-47df-8d12-1ec97ec9707e" containerName="main"
Oct 30 17:23:52.915874 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:52.915753    4433 cpu_manager.go:414] "containerMap: removing container" podUID="eb27ba4a-4385-4f07-a202-f8f3bccb6559" containerName="maas-api"
Oct 30 17:23:52.915874 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:52.915759    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb27ba4a-4385-4f07-a202-f8f3bccb6559" containerName="maas-api"
Oct 30 17:23:52.915874 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:52.915772    4433 cpu_manager.go:414] "containerMap: removing container" podUID="7dded80b-9e71-4566-944c-4d548c59a684" containerName="istio-proxy"
Oct 30 17:23:52.915874 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:52.915779    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dded80b-9e71-4566-944c-4d548c59a684" containerName="istio-proxy"
Oct 30 17:23:52.915874 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:52.915807    4433 cpu_manager.go:414] "containerMap: removing container" podUID="fc12f255-74a5-4c88-b67b-a0c377797e84" containerName="maas-api"
Oct 30 17:23:52.915874 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:52.915814    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc12f255-74a5-4c88-b67b-a0c377797e84" containerName="maas-api"
Oct 30 17:23:52.915874 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:52.915825    4433 cpu_manager.go:414] "containerMap: removing container" podUID="f265af5b-6027-4101-99db-328a61fd85da" containerName="istio-proxy"
Oct 30 17:23:52.915874 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:52.915839    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f265af5b-6027-4101-99db-328a61fd85da" containerName="istio-proxy"
Oct 30 17:23:52.915874 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:52.915848    4433 cpu_manager.go:414] "containerMap: removing container" podUID="c07529e2-9bcd-4ec9-9c98-e0bb48c8effb" containerName="istio-proxy"
Oct 30 17:23:52.915874 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:52.915852    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c07529e2-9bcd-4ec9-9c98-e0bb48c8effb" containerName="istio-proxy"
Oct 30 17:23:52.915874 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:52.915870    4433 cpu_manager.go:414] "containerMap: removing container" podUID="3f6eaf54-07a2-4010-9b79-e82670f15262" containerName="main"
Oct 30 17:23:52.915874 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:52.915875    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f6eaf54-07a2-4010-9b79-e82670f15262" containerName="main"
Oct 30 17:23:52.916672 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:52.916049    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="5420c28a-0d07-47df-8d12-1ec97ec9707e" containerName="main"
Oct 30 17:23:52.916672 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:52.916066    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb27ba4a-4385-4f07-a202-f8f3bccb6559" containerName="maas-api"
Oct 30 17:23:52.916672 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:52.916078    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dded80b-9e71-4566-944c-4d548c59a684" containerName="istio-proxy"
Oct 30 17:23:52.916672 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:52.916090    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc12f255-74a5-4c88-b67b-a0c377797e84" containerName="maas-api"
Oct 30 17:23:52.916672 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:52.916100    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="f265af5b-6027-4101-99db-328a61fd85da" containerName="istio-proxy"
Oct 30 17:23:52.916672 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:52.916108    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="c07529e2-9bcd-4ec9-9c98-e0bb48c8effb" containerName="istio-proxy"
Oct 30 17:23:52.916672 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:52.916118    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f6eaf54-07a2-4010-9b79-e82670f15262" containerName="main"
Oct 30 17:23:52.916672 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:52.916134    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="103bc667-ffe4-4cc3-a66d-fc85ed8a0a51" containerName="istio-proxy"
Oct 30 17:23:52.916672 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:52.916141    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb49dfad-1e85-4b96-9dd8-42374a708643" containerName="istio-proxy"
Oct 30 17:23:52.916672 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:52.916150    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd03e357-fdbb-438b-847e-26ba74b88949" containerName="main"
Oct 30 17:23:52.921204 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:52.921186    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"
Oct 30 17:23:52.935509 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:52.935475    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-78c9c2-openshift-default-dockercfg-2lskh\""
Oct 30 17:23:52.950490 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:52.950466    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"]
Oct 30 17:23:53.021782 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:53.021750    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af923931-b247-4175-b077-658fe2f17c65-istiod-ca-cert\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"
Oct 30 17:23:53.021944 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:53.021865    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af923931-b247-4175-b077-658fe2f17c65-istio-envoy\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"
Oct 30 17:23:53.021944 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:53.021921    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af923931-b247-4175-b077-658fe2f17c65-istio-data\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"
Oct 30 17:23:53.022166 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:53.022133    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af923931-b247-4175-b077-658fe2f17c65-workload-socket\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"
Oct 30 17:23:53.022166 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:53.022171    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af923931-b247-4175-b077-658fe2f17c65-istio-podinfo\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"
Oct 30 17:23:53.022373 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:53.022189    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af923931-b247-4175-b077-658fe2f17c65-credential-socket\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"
Oct 30 17:23:53.022373 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:53.022289    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af923931-b247-4175-b077-658fe2f17c65-istio-token\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"
Oct 30 17:23:53.022373 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:53.022329    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4q9v\" (UniqueName: \"kubernetes.io/projected/af923931-b247-4175-b077-658fe2f17c65-kube-api-access-m4q9v\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"
Oct 30 17:23:53.022570 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:53.022429    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af923931-b247-4175-b077-658fe2f17c65-workload-certs\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"
Oct 30 17:23:53.123460 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:53.123385    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af923931-b247-4175-b077-658fe2f17c65-istio-envoy\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"
Oct 30 17:23:53.123651 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:53.123496    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af923931-b247-4175-b077-658fe2f17c65-istio-data\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"
Oct 30 17:23:53.123702 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:53.123650    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af923931-b247-4175-b077-658fe2f17c65-workload-socket\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"
Oct 30 17:23:53.123702 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:53.123675    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af923931-b247-4175-b077-658fe2f17c65-istio-podinfo\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"
Oct 30 17:23:53.123702 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:53.123697    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af923931-b247-4175-b077-658fe2f17c65-credential-socket\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"
Oct 30 17:23:53.123922 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:53.123867    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af923931-b247-4175-b077-658fe2f17c65-istio-token\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"
Oct 30 17:23:53.124003 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:53.123944    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4q9v\" (UniqueName: \"kubernetes.io/projected/af923931-b247-4175-b077-658fe2f17c65-kube-api-access-m4q9v\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"
Oct 30 17:23:53.124003 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:53.123981    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af923931-b247-4175-b077-658fe2f17c65-workload-certs\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"
Oct 30 17:23:53.124111 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:53.123997    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af923931-b247-4175-b077-658fe2f17c65-istio-data\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"
Oct 30 17:23:53.124111 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:53.124059    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af923931-b247-4175-b077-658fe2f17c65-istiod-ca-cert\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"
Oct 30 17:23:53.124111 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:53.124069    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af923931-b247-4175-b077-658fe2f17c65-workload-socket\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"
Oct 30 17:23:53.124111 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:53.124086    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af923931-b247-4175-b077-658fe2f17c65-credential-socket\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"
Oct 30 17:23:53.124372 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:53.124309    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af923931-b247-4175-b077-658fe2f17c65-workload-certs\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"
Oct 30 17:23:53.124661 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:53.124643    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af923931-b247-4175-b077-658fe2f17c65-istiod-ca-cert\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"
Oct 30 17:23:53.125869 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:53.125849    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af923931-b247-4175-b077-658fe2f17c65-istio-envoy\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"
Oct 30 17:23:53.126146 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:53.126126    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af923931-b247-4175-b077-658fe2f17c65-istio-podinfo\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"
Oct 30 17:23:53.146923 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:53.146892    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af923931-b247-4175-b077-658fe2f17c65-istio-token\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"
Oct 30 17:23:53.147075 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:53.147043    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4q9v\" (UniqueName: \"kubernetes.io/projected/af923931-b247-4175-b077-658fe2f17c65-kube-api-access-m4q9v\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"
Oct 30 17:23:53.238610 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:53.238578    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"
Oct 30 17:23:53.564662 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:53.564634    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"]
Oct 30 17:23:53.566525 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:23:53.566498    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf923931_b247_4175_b077_658fe2f17c65.slice/crio-fb5c4187f29b3e3c0df9bfbaa5b6895b1317c5a2d2ddf5314bdc61c9e8b3e4e2 WatchSource:0}: Error finding container fb5c4187f29b3e3c0df9bfbaa5b6895b1317c5a2d2ddf5314bdc61c9e8b3e4e2: Status 404 returned error can't find the container with id fb5c4187f29b3e3c0df9bfbaa5b6895b1317c5a2d2ddf5314bdc61c9e8b3e4e2
Oct 30 17:23:53.569214 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:53.569179    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:23:53.569290 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:53.569247    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:23:53.569290 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:53.569276    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:23:54.032774 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:54.032732    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5" event={"ID":"af923931-b247-4175-b077-658fe2f17c65","Type":"ContainerStarted","Data":"22e37b96fecc40d49fac7da3c2b051ed42ac6996135c4a0cb012923fa23eddb9"}
Oct 30 17:23:54.033158 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:54.032798    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5" event={"ID":"af923931-b247-4175-b077-658fe2f17c65","Type":"ContainerStarted","Data":"fb5c4187f29b3e3c0df9bfbaa5b6895b1317c5a2d2ddf5314bdc61c9e8b3e4e2"}
Oct 30 17:23:54.080399 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:54.080336    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5" podStartSLOduration=2.080312716 podStartE2EDuration="2.080312716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:23:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:23:54.077329233 +0000 UTC m=+2210.512231458" watchObservedRunningTime="2026-10-30 17:23:54.080312716 +0000 UTC m=+2210.515215013"
Oct 30 17:23:54.239670 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:54.239625    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"
Oct 30 17:23:54.241636 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:54.241598    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 17:23:54.241776 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:54.241664    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5" podUID="af923931-b247-4175-b077-658fe2f17c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 17:23:55.239909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:55.239876    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 17:23:55.240317 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:55.239955    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5" podUID="af923931-b247-4175-b077-658fe2f17c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 17:23:56.239683 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:56.239645    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 17:23:56.239869 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:56.239710    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5" podUID="af923931-b247-4175-b077-658fe2f17c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 17:23:57.003280 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:23:57.003241    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:23:57.239390 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:57.239348    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 17:23:57.239564 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:57.239443    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5" podUID="af923931-b247-4175-b077-658fe2f17c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 17:23:58.239705 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:58.239671    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 17:23:58.240317 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:58.239745    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5" podUID="af923931-b247-4175-b077-658fe2f17c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 17:23:59.239171 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:59.239125    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 17:23:59.239345 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:23:59.239194    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5" podUID="af923931-b247-4175-b077-658fe2f17c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 17:24:00.244427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:00.244384    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"
Oct 30 17:24:00.244911 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:00.244898    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"
Oct 30 17:24:00.245632 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:00.245607    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"
Oct 30 17:24:10.853815 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:10.853780    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"]
Oct 30 17:24:10.859848 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:10.859821    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"
Oct 30 17:24:10.874783 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:10.874755    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-11f480-openshift-default-dockercfg-q54n7\""
Oct 30 17:24:10.890236 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:10.890201    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"]
Oct 30 17:24:10.906759 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:10.906721    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/367df923-d896-440c-8644-86fbc2f6edaf-workload-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"
Oct 30 17:24:10.906909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:10.906765    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/367df923-d896-440c-8644-86fbc2f6edaf-workload-certs\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"
Oct 30 17:24:10.906909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:10.906784    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7bb8\" (UniqueName: \"kubernetes.io/projected/367df923-d896-440c-8644-86fbc2f6edaf-kube-api-access-m7bb8\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"
Oct 30 17:24:10.906909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:10.906850    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/367df923-d896-440c-8644-86fbc2f6edaf-istio-token\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"
Oct 30 17:24:10.906909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:10.906877    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/367df923-d896-440c-8644-86fbc2f6edaf-istio-envoy\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"
Oct 30 17:24:10.907151 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:10.906959    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/367df923-d896-440c-8644-86fbc2f6edaf-istio-data\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"
Oct 30 17:24:10.907151 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:10.907010    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/367df923-d896-440c-8644-86fbc2f6edaf-istiod-ca-cert\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"
Oct 30 17:24:10.907151 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:10.907056    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/367df923-d896-440c-8644-86fbc2f6edaf-credential-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"
Oct 30 17:24:10.907151 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:10.907094    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/367df923-d896-440c-8644-86fbc2f6edaf-istio-podinfo\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"
Oct 30 17:24:11.008100 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:11.008064    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/367df923-d896-440c-8644-86fbc2f6edaf-istio-data\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"
Oct 30 17:24:11.008284 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:11.008152    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/367df923-d896-440c-8644-86fbc2f6edaf-istiod-ca-cert\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"
Oct 30 17:24:11.008284 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:11.008218    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/367df923-d896-440c-8644-86fbc2f6edaf-credential-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"
Oct 30 17:24:11.008284 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:11.008264    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/367df923-d896-440c-8644-86fbc2f6edaf-istio-podinfo\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"
Oct 30 17:24:11.008521 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:11.008311    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/367df923-d896-440c-8644-86fbc2f6edaf-workload-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"
Oct 30 17:24:11.008521 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:11.008342    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/367df923-d896-440c-8644-86fbc2f6edaf-workload-certs\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"
Oct 30 17:24:11.008521 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:11.008366    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7bb8\" (UniqueName: \"kubernetes.io/projected/367df923-d896-440c-8644-86fbc2f6edaf-kube-api-access-m7bb8\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"
Oct 30 17:24:11.008521 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:11.008470    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/367df923-d896-440c-8644-86fbc2f6edaf-istio-token\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"
Oct 30 17:24:11.008521 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:11.008513    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/367df923-d896-440c-8644-86fbc2f6edaf-istio-envoy\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"
Oct 30 17:24:11.008768 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:11.008588    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/367df923-d896-440c-8644-86fbc2f6edaf-istio-data\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"
Oct 30 17:24:11.008850 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:11.008820    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/367df923-d896-440c-8644-86fbc2f6edaf-credential-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"
Oct 30 17:24:11.008971 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:11.008941    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/367df923-d896-440c-8644-86fbc2f6edaf-workload-certs\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"
Oct 30 17:24:11.009130 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:11.009093    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/367df923-d896-440c-8644-86fbc2f6edaf-workload-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"
Oct 30 17:24:11.009272 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:11.009250    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/367df923-d896-440c-8644-86fbc2f6edaf-istiod-ca-cert\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"
Oct 30 17:24:11.010742 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:11.010715    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/367df923-d896-440c-8644-86fbc2f6edaf-istio-envoy\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"
Oct 30 17:24:11.010873 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:11.010856    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/367df923-d896-440c-8644-86fbc2f6edaf-istio-podinfo\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"
Oct 30 17:24:11.031619 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:11.031594    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7bb8\" (UniqueName: \"kubernetes.io/projected/367df923-d896-440c-8644-86fbc2f6edaf-kube-api-access-m7bb8\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"
Oct 30 17:24:11.031732 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:11.031679    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/367df923-d896-440c-8644-86fbc2f6edaf-istio-token\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"
Oct 30 17:24:11.177525 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:11.177492    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"
Oct 30 17:24:11.507546 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:11.507518    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"]
Oct 30 17:24:11.512619 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:24:11.512575    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod367df923_d896_440c_8644_86fbc2f6edaf.slice/crio-c967f0daf90ddf6a1e9b8ad5648526ae812b543b14215e2bc46dbcad46ed1f7e WatchSource:0}: Error finding container c967f0daf90ddf6a1e9b8ad5648526ae812b543b14215e2bc46dbcad46ed1f7e: Status 404 returned error can't find the container with id c967f0daf90ddf6a1e9b8ad5648526ae812b543b14215e2bc46dbcad46ed1f7e
Oct 30 17:24:11.515496 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:11.515456    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:24:11.515593 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:11.515541    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:24:11.515593 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:11.515579    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:24:12.174057 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:12.174023    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc" event={"ID":"367df923-d896-440c-8644-86fbc2f6edaf","Type":"ContainerStarted","Data":"22582ef841c0ce5175a70e6194da83c92429580924eb483c6c05119dee717273"}
Oct 30 17:24:12.174529 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:12.174074    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc" event={"ID":"367df923-d896-440c-8644-86fbc2f6edaf","Type":"ContainerStarted","Data":"c967f0daf90ddf6a1e9b8ad5648526ae812b543b14215e2bc46dbcad46ed1f7e"}
Oct 30 17:24:12.178371 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:12.178349    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"
Oct 30 17:24:12.179696 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:12.179673    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 17:24:12.179784 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:12.179726    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc" podUID="367df923-d896-440c-8644-86fbc2f6edaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 17:24:12.185255 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:12.185231    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"]
Oct 30 17:24:12.185488 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:12.185453    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9" podUID="41ff63dc-b547-4143-adb5-d904bc6e91d8" containerName="istio-proxy" containerID="cri-o://9793b76780896f189144a1b76124914e6bffebd698d34db7e905ed9439eff0b2" gracePeriod=30
Oct 30 17:24:12.232655 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:12.232604    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc" podStartSLOduration=2.232589188 podStartE2EDuration="2.232589188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:12.22939925 +0000 UTC m=+2228.664301468" watchObservedRunningTime="2026-10-30 17:24:12.232589188 +0000 UTC m=+2228.667491394"
Oct 30 17:24:12.250593 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:12.250560    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"]
Oct 30 17:24:12.250769 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:12.250742    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf" podUID="be1ee2aa-dabc-4fab-b1d6-24c01758a780" containerName="istio-proxy" containerID="cri-o://96c9190092fb55255a27ea764d965acbb55a32c6598f1227ddb222d12784f128" gracePeriod=30
Oct 30 17:24:13.185147 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:13.183863    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 17:24:13.185147 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:13.183939    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc" podUID="367df923-d896-440c-8644-86fbc2f6edaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 17:24:14.178449 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:14.178368    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 17:24:14.178639 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:14.178486    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc" podUID="367df923-d896-440c-8644-86fbc2f6edaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 17:24:15.178564 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:15.178522    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 17:24:15.179052 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:15.178585    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc" podUID="367df923-d896-440c-8644-86fbc2f6edaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 17:24:16.186428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:16.183030    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 17:24:16.186428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:16.183109    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc" podUID="367df923-d896-440c-8644-86fbc2f6edaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 17:24:16.244170 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:16.244134    4433 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:24:16.244339 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:16.244199    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9" podUID="41ff63dc-b547-4143-adb5-d904bc6e91d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:24:16.815028 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:16.814984    4433 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:24:16.815208 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:16.815061    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf" podUID="be1ee2aa-dabc-4fab-b1d6-24c01758a780" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:24:17.178488 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.178448    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 17:24:17.178679 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.178534    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc" podUID="367df923-d896-440c-8644-86fbc2f6edaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 17:24:17.628598 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.628570    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"
Oct 30 17:24:17.647908 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.647864    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"
Oct 30 17:24:17.791220 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.791118    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41ff63dc-b547-4143-adb5-d904bc6e91d8-istio-token\" (UniqueName: \"kubernetes.io/projected/41ff63dc-b547-4143-adb5-d904bc6e91d8-istio-token\") pod \"41ff63dc-b547-4143-adb5-d904bc6e91d8\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") "
Oct 30 17:24:17.791220 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.791163    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41ff63dc-b547-4143-adb5-d904bc6e91d8-kube-api-access-nrprp\" (UniqueName: \"kubernetes.io/projected/41ff63dc-b547-4143-adb5-d904bc6e91d8-kube-api-access-nrprp\") pod \"41ff63dc-b547-4143-adb5-d904bc6e91d8\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") "
Oct 30 17:24:17.791493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.791258    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41ff63dc-b547-4143-adb5-d904bc6e91d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ff63dc-b547-4143-adb5-d904bc6e91d8-workload-socket\") pod \"41ff63dc-b547-4143-adb5-d904bc6e91d8\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") "
Oct 30 17:24:17.791493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.791295    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be1ee2aa-dabc-4fab-b1d6-24c01758a780-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be1ee2aa-dabc-4fab-b1d6-24c01758a780-istio-data\") pod \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") "
Oct 30 17:24:17.791493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.791312    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be1ee2aa-dabc-4fab-b1d6-24c01758a780-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be1ee2aa-dabc-4fab-b1d6-24c01758a780-istio-envoy\") pod \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") "
Oct 30 17:24:17.791493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.791344    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be1ee2aa-dabc-4fab-b1d6-24c01758a780-kube-api-access-sgv7j\" (UniqueName: \"kubernetes.io/projected/be1ee2aa-dabc-4fab-b1d6-24c01758a780-kube-api-access-sgv7j\") pod \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") "
Oct 30 17:24:17.791493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.791366    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be1ee2aa-dabc-4fab-b1d6-24c01758a780-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1ee2aa-dabc-4fab-b1d6-24c01758a780-workload-socket\") pod \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") "
Oct 30 17:24:17.791493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.791389    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41ff63dc-b547-4143-adb5-d904bc6e91d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41ff63dc-b547-4143-adb5-d904bc6e91d8-istiod-ca-cert\") pod \"41ff63dc-b547-4143-adb5-d904bc6e91d8\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") "
Oct 30 17:24:17.791493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.791443    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41ff63dc-b547-4143-adb5-d904bc6e91d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41ff63dc-b547-4143-adb5-d904bc6e91d8-istio-podinfo\") pod \"41ff63dc-b547-4143-adb5-d904bc6e91d8\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") "
Oct 30 17:24:17.791493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.791475    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be1ee2aa-dabc-4fab-b1d6-24c01758a780-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1ee2aa-dabc-4fab-b1d6-24c01758a780-credential-socket\") pod \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") "
Oct 30 17:24:17.791923 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.791529    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41ff63dc-b547-4143-adb5-d904bc6e91d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41ff63dc-b547-4143-adb5-d904bc6e91d8-workload-certs\") pod \"41ff63dc-b547-4143-adb5-d904bc6e91d8\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") "
Oct 30 17:24:17.791923 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.791559    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be1ee2aa-dabc-4fab-b1d6-24c01758a780-istio-token\" (UniqueName: \"kubernetes.io/projected/be1ee2aa-dabc-4fab-b1d6-24c01758a780-istio-token\") pod \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") "
Oct 30 17:24:17.791923 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.791611    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41ff63dc-b547-4143-adb5-d904bc6e91d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41ff63dc-b547-4143-adb5-d904bc6e91d8-istio-data\") pod \"41ff63dc-b547-4143-adb5-d904bc6e91d8\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") "
Oct 30 17:24:17.791923 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.791713    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be1ee2aa-dabc-4fab-b1d6-24c01758a780-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be1ee2aa-dabc-4fab-b1d6-24c01758a780-workload-certs\") pod \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") "
Oct 30 17:24:17.791923 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.791748    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be1ee2aa-dabc-4fab-b1d6-24c01758a780-istio-data" pod "be1ee2aa-dabc-4fab-b1d6-24c01758a780" (UID: "be1ee2aa-dabc-4fab-b1d6-24c01758a780"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:17.791923 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.791764    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be1ee2aa-dabc-4fab-b1d6-24c01758a780-workload-socket" pod "be1ee2aa-dabc-4fab-b1d6-24c01758a780" (UID: "be1ee2aa-dabc-4fab-b1d6-24c01758a780"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:17.791923 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.791771    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be1ee2aa-dabc-4fab-b1d6-24c01758a780-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be1ee2aa-dabc-4fab-b1d6-24c01758a780-istio-podinfo\") pod \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") "
Oct 30 17:24:17.791923 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.791840    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41ff63dc-b547-4143-adb5-d904bc6e91d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ff63dc-b547-4143-adb5-d904bc6e91d8-credential-socket\") pod \"41ff63dc-b547-4143-adb5-d904bc6e91d8\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") "
Oct 30 17:24:17.792323 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.791958    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be1ee2aa-dabc-4fab-b1d6-24c01758a780-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be1ee2aa-dabc-4fab-b1d6-24c01758a780-istiod-ca-cert\") pod \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\" (UID: \"be1ee2aa-dabc-4fab-b1d6-24c01758a780\") "
Oct 30 17:24:17.792323 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.792038    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41ff63dc-b547-4143-adb5-d904bc6e91d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41ff63dc-b547-4143-adb5-d904bc6e91d8-istio-envoy\") pod \"41ff63dc-b547-4143-adb5-d904bc6e91d8\" (UID: \"41ff63dc-b547-4143-adb5-d904bc6e91d8\") "
Oct 30 17:24:17.792323 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.792213    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ff63dc-b547-4143-adb5-d904bc6e91d8-credential-socket" pod "41ff63dc-b547-4143-adb5-d904bc6e91d8" (UID: "41ff63dc-b547-4143-adb5-d904bc6e91d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:17.792491 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.792391    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ff63dc-b547-4143-adb5-d904bc6e91d8-workload-socket" pod "41ff63dc-b547-4143-adb5-d904bc6e91d8" (UID: "41ff63dc-b547-4143-adb5-d904bc6e91d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:17.792625 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.792601    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ff63dc-b547-4143-adb5-d904bc6e91d8-workload-certs" pod "41ff63dc-b547-4143-adb5-d904bc6e91d8" (UID: "41ff63dc-b547-4143-adb5-d904bc6e91d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:17.792979 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.792869    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ff63dc-b547-4143-adb5-d904bc6e91d8-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:24:17.792979 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.792892    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be1ee2aa-dabc-4fab-b1d6-24c01758a780-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:24:17.792979 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.792905    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1ee2aa-dabc-4fab-b1d6-24c01758a780-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:24:17.792979 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.792913    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41ff63dc-b547-4143-adb5-d904bc6e91d8-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:24:17.792979 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.792922    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ff63dc-b547-4143-adb5-d904bc6e91d8-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:24:17.792979 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.792939    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be1ee2aa-dabc-4fab-b1d6-24c01758a780-istiod-ca-cert" pod "be1ee2aa-dabc-4fab-b1d6-24c01758a780" (UID: "be1ee2aa-dabc-4fab-b1d6-24c01758a780"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:24:17.792979 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.792962    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be1ee2aa-dabc-4fab-b1d6-24c01758a780-credential-socket" pod "be1ee2aa-dabc-4fab-b1d6-24c01758a780" (UID: "be1ee2aa-dabc-4fab-b1d6-24c01758a780"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:17.793503 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.793146    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ff63dc-b547-4143-adb5-d904bc6e91d8-istio-data" pod "41ff63dc-b547-4143-adb5-d904bc6e91d8" (UID: "41ff63dc-b547-4143-adb5-d904bc6e91d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:17.793503 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.793319    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41ff63dc-b547-4143-adb5-d904bc6e91d8-istiod-ca-cert" pod "41ff63dc-b547-4143-adb5-d904bc6e91d8" (UID: "41ff63dc-b547-4143-adb5-d904bc6e91d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:24:17.793619 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.793594    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be1ee2aa-dabc-4fab-b1d6-24c01758a780-workload-certs" pod "be1ee2aa-dabc-4fab-b1d6-24c01758a780" (UID: "be1ee2aa-dabc-4fab-b1d6-24c01758a780"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:17.796666 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.796352    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be1ee2aa-dabc-4fab-b1d6-24c01758a780-istio-podinfo" pod "be1ee2aa-dabc-4fab-b1d6-24c01758a780" (UID: "be1ee2aa-dabc-4fab-b1d6-24c01758a780"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:24:17.796666 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.796420    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be1ee2aa-dabc-4fab-b1d6-24c01758a780-kube-api-access-sgv7j" pod "be1ee2aa-dabc-4fab-b1d6-24c01758a780" (UID: "be1ee2aa-dabc-4fab-b1d6-24c01758a780"). InnerVolumeSpecName "kube-api-access-sgv7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:24:17.798104 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.798079    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41ff63dc-b547-4143-adb5-d904bc6e91d8-istio-token" pod "41ff63dc-b547-4143-adb5-d904bc6e91d8" (UID: "41ff63dc-b547-4143-adb5-d904bc6e91d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:24:17.798213 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.798154    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41ff63dc-b547-4143-adb5-d904bc6e91d8-kube-api-access-nrprp" pod "41ff63dc-b547-4143-adb5-d904bc6e91d8" (UID: "41ff63dc-b547-4143-adb5-d904bc6e91d8"). InnerVolumeSpecName "kube-api-access-nrprp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:24:17.798213 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.798204    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be1ee2aa-dabc-4fab-b1d6-24c01758a780-istio-envoy" pod "be1ee2aa-dabc-4fab-b1d6-24c01758a780" (UID: "be1ee2aa-dabc-4fab-b1d6-24c01758a780"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:17.798326 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.798256    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be1ee2aa-dabc-4fab-b1d6-24c01758a780-istio-token" pod "be1ee2aa-dabc-4fab-b1d6-24c01758a780" (UID: "be1ee2aa-dabc-4fab-b1d6-24c01758a780"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:24:17.799307 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.799276    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41ff63dc-b547-4143-adb5-d904bc6e91d8-istio-podinfo" pod "41ff63dc-b547-4143-adb5-d904bc6e91d8" (UID: "41ff63dc-b547-4143-adb5-d904bc6e91d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:24:17.799307 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.799299    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ff63dc-b547-4143-adb5-d904bc6e91d8-istio-envoy" pod "41ff63dc-b547-4143-adb5-d904bc6e91d8" (UID: "41ff63dc-b547-4143-adb5-d904bc6e91d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:17.894016 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.893977    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41ff63dc-b547-4143-adb5-d904bc6e91d8-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:24:17.894016 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.894011    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be1ee2aa-dabc-4fab-b1d6-24c01758a780-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:24:17.894016 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.894020    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be1ee2aa-dabc-4fab-b1d6-24c01758a780-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:24:17.894257 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.894028    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be1ee2aa-dabc-4fab-b1d6-24c01758a780-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:24:17.894257 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.894036    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41ff63dc-b547-4143-adb5-d904bc6e91d8-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:24:17.894257 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.894044    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41ff63dc-b547-4143-adb5-d904bc6e91d8-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:24:17.894257 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.894052    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrprp\" (UniqueName: \"kubernetes.io/projected/41ff63dc-b547-4143-adb5-d904bc6e91d8-kube-api-access-nrprp\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:24:17.894257 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.894061    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be1ee2aa-dabc-4fab-b1d6-24c01758a780-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:24:17.894257 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.894073    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgv7j\" (UniqueName: \"kubernetes.io/projected/be1ee2aa-dabc-4fab-b1d6-24c01758a780-kube-api-access-sgv7j\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:24:17.894257 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.894085    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41ff63dc-b547-4143-adb5-d904bc6e91d8-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:24:17.894257 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.894099    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41ff63dc-b547-4143-adb5-d904bc6e91d8-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:24:17.894257 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.894111    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1ee2aa-dabc-4fab-b1d6-24c01758a780-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:24:17.894257 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:17.894125    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be1ee2aa-dabc-4fab-b1d6-24c01758a780-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:24:18.178373 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:18.178336    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 17:24:18.178577 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:18.178443    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc" podUID="367df923-d896-440c-8644-86fbc2f6edaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 17:24:18.232844 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:18.232799    4433 generic.go:356] "Generic (PLEG): container finished" podID="41ff63dc-b547-4143-adb5-d904bc6e91d8" containerID="9793b76780896f189144a1b76124914e6bffebd698d34db7e905ed9439eff0b2" exitCode=0
Oct 30 17:24:18.233034 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:18.232876    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"
Oct 30 17:24:18.233034 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:18.232873    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9" event={"ID":"41ff63dc-b547-4143-adb5-d904bc6e91d8","Type":"ContainerDied","Data":"9793b76780896f189144a1b76124914e6bffebd698d34db7e905ed9439eff0b2"}
Oct 30 17:24:18.233034 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:18.232986    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9" event={"ID":"41ff63dc-b547-4143-adb5-d904bc6e91d8","Type":"ContainerDied","Data":"68c7814d00e519dfbfe06fa058e0a588ebd9c5ecfa48dd5302bcf01375f71932"}
Oct 30 17:24:18.233034 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:18.233022    4433 scope.go:122] "RemoveContainer" containerID="9793b76780896f189144a1b76124914e6bffebd698d34db7e905ed9439eff0b2"
Oct 30 17:24:18.234623 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:18.234598    4433 generic.go:356] "Generic (PLEG): container finished" podID="be1ee2aa-dabc-4fab-b1d6-24c01758a780" containerID="96c9190092fb55255a27ea764d965acbb55a32c6598f1227ddb222d12784f128" exitCode=0
Oct 30 17:24:18.234829 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:18.234654    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"
Oct 30 17:24:18.234829 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:18.234686    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf" event={"ID":"be1ee2aa-dabc-4fab-b1d6-24c01758a780","Type":"ContainerDied","Data":"96c9190092fb55255a27ea764d965acbb55a32c6598f1227ddb222d12784f128"}
Oct 30 17:24:18.234829 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:18.234732    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf" event={"ID":"be1ee2aa-dabc-4fab-b1d6-24c01758a780","Type":"ContainerDied","Data":"20ccb2a3f431f39606336bc12c543c6441e8ee4df5caeebe508c04959ec3f528"}
Oct 30 17:24:18.246689 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:18.246653    4433 scope.go:122] "RemoveContainer" containerID="9793b76780896f189144a1b76124914e6bffebd698d34db7e905ed9439eff0b2"
Oct 30 17:24:18.247187 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:24:18.247161    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9793b76780896f189144a1b76124914e6bffebd698d34db7e905ed9439eff0b2\": container with ID starting with 9793b76780896f189144a1b76124914e6bffebd698d34db7e905ed9439eff0b2 not found: ID does not exist" containerID="9793b76780896f189144a1b76124914e6bffebd698d34db7e905ed9439eff0b2"
Oct 30 17:24:18.247309 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:18.247200    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9793b76780896f189144a1b76124914e6bffebd698d34db7e905ed9439eff0b2"} err="failed to get container status \"9793b76780896f189144a1b76124914e6bffebd698d34db7e905ed9439eff0b2\": rpc error: code = NotFound desc = could not find container \"9793b76780896f189144a1b76124914e6bffebd698d34db7e905ed9439eff0b2\": container with ID starting with 9793b76780896f189144a1b76124914e6bffebd698d34db7e905ed9439eff0b2 not found: ID does not exist"
Oct 30 17:24:18.247309 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:18.247224    4433 scope.go:122] "RemoveContainer" containerID="96c9190092fb55255a27ea764d965acbb55a32c6598f1227ddb222d12784f128"
Oct 30 17:24:18.257304 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:18.257279    4433 scope.go:122] "RemoveContainer" containerID="96c9190092fb55255a27ea764d965acbb55a32c6598f1227ddb222d12784f128"
Oct 30 17:24:18.257664 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:24:18.257646    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96c9190092fb55255a27ea764d965acbb55a32c6598f1227ddb222d12784f128\": container with ID starting with 96c9190092fb55255a27ea764d965acbb55a32c6598f1227ddb222d12784f128 not found: ID does not exist" containerID="96c9190092fb55255a27ea764d965acbb55a32c6598f1227ddb222d12784f128"
Oct 30 17:24:18.257731 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:18.257673    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96c9190092fb55255a27ea764d965acbb55a32c6598f1227ddb222d12784f128"} err="failed to get container status \"96c9190092fb55255a27ea764d965acbb55a32c6598f1227ddb222d12784f128\": rpc error: code = NotFound desc = could not find container \"96c9190092fb55255a27ea764d965acbb55a32c6598f1227ddb222d12784f128\": container with ID starting with 96c9190092fb55255a27ea764d965acbb55a32c6598f1227ddb222d12784f128 not found: ID does not exist"
Oct 30 17:24:18.287526 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:18.287490    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"]
Oct 30 17:24:18.298302 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:18.298262    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-nfph9"]
Oct 30 17:24:18.324396 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:18.324354    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"]
Oct 30 17:24:18.331888 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:18.331848    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-7ztcf"]
Oct 30 17:24:19.183542 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:19.183512    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"
Oct 30 17:24:19.184007 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:19.183943    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"
Oct 30 17:24:19.184643 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:19.184624    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"
Oct 30 17:24:19.217980 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:19.217941    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg"]
Oct 30 17:24:19.218193 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:19.218167    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg" podUID="7a8f95b8-4923-4d7c-8b6b-2e65231eeb09" containerName="maas-api" containerID="cri-o://e1a2d5fa9d4ed90cd63091c2573e57241dda07d00eac28634dce726330efa8dc" gracePeriod=30
Oct 30 17:24:19.665087 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:19.664926    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg"
Oct 30 17:24:19.695935 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:19.695884    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41ff63dc-b547-4143-adb5-d904bc6e91d8" path="/var/lib/kubelet/pods/41ff63dc-b547-4143-adb5-d904bc6e91d8/volumes"
Oct 30 17:24:19.696557 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:19.696537    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be1ee2aa-dabc-4fab-b1d6-24c01758a780" path="/var/lib/kubelet/pods/be1ee2aa-dabc-4fab-b1d6-24c01758a780/volumes"
Oct 30 17:24:19.849469 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:19.849432    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a8f95b8-4923-4d7c-8b6b-2e65231eeb09-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a8f95b8-4923-4d7c-8b6b-2e65231eeb09-metrics-certs\") pod \"7a8f95b8-4923-4d7c-8b6b-2e65231eeb09\" (UID: \"7a8f95b8-4923-4d7c-8b6b-2e65231eeb09\") "
Oct 30 17:24:19.849656 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:19.849490    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a8f95b8-4923-4d7c-8b6b-2e65231eeb09-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a8f95b8-4923-4d7c-8b6b-2e65231eeb09-openshift-service-ca\") pod \"7a8f95b8-4923-4d7c-8b6b-2e65231eeb09\" (UID: \"7a8f95b8-4923-4d7c-8b6b-2e65231eeb09\") "
Oct 30 17:24:19.849656 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:19.849571    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a8f95b8-4923-4d7c-8b6b-2e65231eeb09-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a8f95b8-4923-4d7c-8b6b-2e65231eeb09-maas-api-tls\") pod \"7a8f95b8-4923-4d7c-8b6b-2e65231eeb09\" (UID: \"7a8f95b8-4923-4d7c-8b6b-2e65231eeb09\") "
Oct 30 17:24:19.849805 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:19.849663    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a8f95b8-4923-4d7c-8b6b-2e65231eeb09-kube-api-access-gj8lp\" (UniqueName: \"kubernetes.io/projected/7a8f95b8-4923-4d7c-8b6b-2e65231eeb09-kube-api-access-gj8lp\") pod \"7a8f95b8-4923-4d7c-8b6b-2e65231eeb09\" (UID: \"7a8f95b8-4923-4d7c-8b6b-2e65231eeb09\") "
Oct 30 17:24:19.850504 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:19.850367    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a8f95b8-4923-4d7c-8b6b-2e65231eeb09-openshift-service-ca" pod "7a8f95b8-4923-4d7c-8b6b-2e65231eeb09" (UID: "7a8f95b8-4923-4d7c-8b6b-2e65231eeb09"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:24:19.857527 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:19.853322    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a8f95b8-4923-4d7c-8b6b-2e65231eeb09-metrics-certs" pod "7a8f95b8-4923-4d7c-8b6b-2e65231eeb09" (UID: "7a8f95b8-4923-4d7c-8b6b-2e65231eeb09"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:24:19.857527 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:19.855555    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a8f95b8-4923-4d7c-8b6b-2e65231eeb09-maas-api-tls" pod "7a8f95b8-4923-4d7c-8b6b-2e65231eeb09" (UID: "7a8f95b8-4923-4d7c-8b6b-2e65231eeb09"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:24:19.857527 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:19.857227    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a8f95b8-4923-4d7c-8b6b-2e65231eeb09-kube-api-access-gj8lp" pod "7a8f95b8-4923-4d7c-8b6b-2e65231eeb09" (UID: "7a8f95b8-4923-4d7c-8b6b-2e65231eeb09"). InnerVolumeSpecName "kube-api-access-gj8lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:24:19.952105 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:19.952066    4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a8f95b8-4923-4d7c-8b6b-2e65231eeb09-metrics-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:24:19.952105 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:19.952102    4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a8f95b8-4923-4d7c-8b6b-2e65231eeb09-openshift-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:24:19.952105 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:19.952112    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a8f95b8-4923-4d7c-8b6b-2e65231eeb09-maas-api-tls\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:24:19.952345 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:19.952120    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj8lp\" (UniqueName: \"kubernetes.io/projected/7a8f95b8-4923-4d7c-8b6b-2e65231eeb09-kube-api-access-gj8lp\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:24:20.264773 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:20.264738    4433 generic.go:356] "Generic (PLEG): container finished" podID="7a8f95b8-4923-4d7c-8b6b-2e65231eeb09" containerID="e1a2d5fa9d4ed90cd63091c2573e57241dda07d00eac28634dce726330efa8dc" exitCode=0
Oct 30 17:24:20.265220 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:20.264787    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg"
Oct 30 17:24:20.265220 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:20.264825    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg" event={"ID":"7a8f95b8-4923-4d7c-8b6b-2e65231eeb09","Type":"ContainerDied","Data":"e1a2d5fa9d4ed90cd63091c2573e57241dda07d00eac28634dce726330efa8dc"}
Oct 30 17:24:20.265220 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:20.264879    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg" event={"ID":"7a8f95b8-4923-4d7c-8b6b-2e65231eeb09","Type":"ContainerDied","Data":"1e4c2da0aadf3ebd42698bebc799d08a018df81c1dabf47fba39d04e2486cb1e"}
Oct 30 17:24:20.265220 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:20.264901    4433 scope.go:122] "RemoveContainer" containerID="e1a2d5fa9d4ed90cd63091c2573e57241dda07d00eac28634dce726330efa8dc"
Oct 30 17:24:20.277082 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:20.277055    4433 scope.go:122] "RemoveContainer" containerID="e1a2d5fa9d4ed90cd63091c2573e57241dda07d00eac28634dce726330efa8dc"
Oct 30 17:24:20.277399 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:24:20.277380    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1a2d5fa9d4ed90cd63091c2573e57241dda07d00eac28634dce726330efa8dc\": container with ID starting with e1a2d5fa9d4ed90cd63091c2573e57241dda07d00eac28634dce726330efa8dc not found: ID does not exist" containerID="e1a2d5fa9d4ed90cd63091c2573e57241dda07d00eac28634dce726330efa8dc"
Oct 30 17:24:20.277489 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:20.277427    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1a2d5fa9d4ed90cd63091c2573e57241dda07d00eac28634dce726330efa8dc"} err="failed to get container status \"e1a2d5fa9d4ed90cd63091c2573e57241dda07d00eac28634dce726330efa8dc\": rpc error: code = NotFound desc = could not find container \"e1a2d5fa9d4ed90cd63091c2573e57241dda07d00eac28634dce726330efa8dc\": container with ID starting with e1a2d5fa9d4ed90cd63091c2573e57241dda07d00eac28634dce726330efa8dc not found: ID does not exist"
Oct 30 17:24:20.319417 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:20.319370    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg"]
Oct 30 17:24:20.330434 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:20.330378    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dd5c8f-6cfc47564-lccqg"]
Oct 30 17:24:21.695832 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:21.695795    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a8f95b8-4923-4d7c-8b6b-2e65231eeb09" path="/var/lib/kubelet/pods/7a8f95b8-4923-4d7c-8b6b-2e65231eeb09/volumes"
Oct 30 17:24:25.769725 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.769688    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"]
Oct 30 17:24:25.770351 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.770325    4433 cpu_manager.go:414] "containerMap: removing container" podUID="be1ee2aa-dabc-4fab-b1d6-24c01758a780" containerName="istio-proxy"
Oct 30 17:24:25.770351 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.770350    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be1ee2aa-dabc-4fab-b1d6-24c01758a780" containerName="istio-proxy"
Oct 30 17:24:25.770607 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.770463    4433 cpu_manager.go:414] "containerMap: removing container" podUID="41ff63dc-b547-4143-adb5-d904bc6e91d8" containerName="istio-proxy"
Oct 30 17:24:25.770607 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.770471    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41ff63dc-b547-4143-adb5-d904bc6e91d8" containerName="istio-proxy"
Oct 30 17:24:25.770607 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.770492    4433 cpu_manager.go:414] "containerMap: removing container" podUID="7a8f95b8-4923-4d7c-8b6b-2e65231eeb09" containerName="maas-api"
Oct 30 17:24:25.770607 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.770499    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a8f95b8-4923-4d7c-8b6b-2e65231eeb09" containerName="maas-api"
Oct 30 17:24:25.770781 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.770639    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="be1ee2aa-dabc-4fab-b1d6-24c01758a780" containerName="istio-proxy"
Oct 30 17:24:25.770781 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.770665    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="41ff63dc-b547-4143-adb5-d904bc6e91d8" containerName="istio-proxy"
Oct 30 17:24:25.770781 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.770672    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a8f95b8-4923-4d7c-8b6b-2e65231eeb09" containerName="maas-api"
Oct 30 17:24:25.775790 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.775765    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"
Oct 30 17:24:25.796638 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.796609    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"]
Oct 30 17:24:25.800723 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.800688    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"
Oct 30 17:24:25.808442 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.808377    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"]
Oct 30 17:24:25.835506 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.835476    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"]
Oct 30 17:24:25.862237 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.862204    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-istio-token\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"
Oct 30 17:24:25.862428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.862258    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-istio-podinfo\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"
Oct 30 17:24:25.862428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.862336    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-workload-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"
Oct 30 17:24:25.862575 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.862441    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-istio-envoy\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"
Oct 30 17:24:25.862575 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.862549    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-istiod-ca-cert\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"
Oct 30 17:24:25.862686 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.862624    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-istiod-ca-cert\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"
Oct 30 17:24:25.862743 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.862683    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-credential-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"
Oct 30 17:24:25.862743 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.862722    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-workload-certs\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"
Oct 30 17:24:25.862820 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.862753    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-istio-data\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"
Oct 30 17:24:25.862863 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.862817    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-istio-token\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"
Oct 30 17:24:25.862863 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.862852    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-workload-certs\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"
Oct 30 17:24:25.862957 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.862937    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-istio-data\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"
Oct 30 17:24:25.863086 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.862990    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-credential-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"
Oct 30 17:24:25.863086 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.863062    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s84fp\" (UniqueName: \"kubernetes.io/projected/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-kube-api-access-s84fp\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"
Oct 30 17:24:25.863197 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.863091    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-istio-envoy\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"
Oct 30 17:24:25.863197 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.863106    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-istio-podinfo\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"
Oct 30 17:24:25.863197 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.863152    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-workload-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"
Oct 30 17:24:25.863197 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.863172    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsn78\" (UniqueName: \"kubernetes.io/projected/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-kube-api-access-dsn78\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"
Oct 30 17:24:25.964931 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.964896    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-istio-data\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"
Oct 30 17:24:25.964931 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.964935    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-istio-token\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"
Oct 30 17:24:25.965221 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.964960    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-workload-certs\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"
Oct 30 17:24:25.965221 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.964989    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-istio-data\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"
Oct 30 17:24:25.965221 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.965027    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-credential-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"
Oct 30 17:24:25.965221 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.965102    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s84fp\" (UniqueName: \"kubernetes.io/projected/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-kube-api-access-s84fp\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"
Oct 30 17:24:25.965221 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.965146    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-istio-envoy\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"
Oct 30 17:24:25.965221 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.965168    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-istio-podinfo\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"
Oct 30 17:24:25.966514 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.965229    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-workload-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"
Oct 30 17:24:25.966514 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.965263    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsn78\" (UniqueName: \"kubernetes.io/projected/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-kube-api-access-dsn78\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"
Oct 30 17:24:25.966514 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.965318    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-istio-token\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"
Oct 30 17:24:25.966514 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.965428    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-istio-data\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"
Oct 30 17:24:25.966514 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.965473    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-workload-certs\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"
Oct 30 17:24:25.966514 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.965484    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-istio-podinfo\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"
Oct 30 17:24:25.966514 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.965534    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-workload-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"
Oct 30 17:24:25.966514 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.965568    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-istio-envoy\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"
Oct 30 17:24:25.966514 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.965599    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-credential-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"
Oct 30 17:24:25.966514 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.965667    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-istiod-ca-cert\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"
Oct 30 17:24:25.966514 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.965737    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-istiod-ca-cert\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"
Oct 30 17:24:25.966514 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.965770    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-workload-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"
Oct 30 17:24:25.966514 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.965789    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-credential-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"
Oct 30 17:24:25.966514 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.965839    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-workload-certs\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"
Oct 30 17:24:25.966514 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.966032    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-credential-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"
Oct 30 17:24:25.966514 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.966181    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-workload-certs\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"
Oct 30 17:24:25.968114 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.967212    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-istiod-ca-cert\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"
Oct 30 17:24:25.968114 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.967535    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-istio-data\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"
Oct 30 17:24:25.968114 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.967846    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-istio-envoy\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"
Oct 30 17:24:25.968279 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.968130    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-istio-envoy\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"
Oct 30 17:24:25.968379 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.968356    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-istio-podinfo\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"
Oct 30 17:24:25.968552 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.968529    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-istio-podinfo\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"
Oct 30 17:24:25.968854 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.968836    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-istiod-ca-cert\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"
Oct 30 17:24:25.968972 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.968952    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-workload-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"
Oct 30 17:24:25.987989 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.987949    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-istio-token\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"
Oct 30 17:24:25.988524 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.988499    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsn78\" (UniqueName: \"kubernetes.io/projected/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-kube-api-access-dsn78\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"
Oct 30 17:24:25.988613 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.988586    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-istio-token\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"
Oct 30 17:24:25.989001 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:25.988979    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s84fp\" (UniqueName: \"kubernetes.io/projected/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-kube-api-access-s84fp\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"
Oct 30 17:24:26.091849 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:26.091760    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"
Oct 30 17:24:26.131686 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:26.131651    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"
Oct 30 17:24:26.416451 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:26.416420    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"]
Oct 30 17:24:26.418644 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:24:26.418599    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce54e5a_f0d0_4f76_9f66_ad577aca3fe9.slice/crio-2d7cc86de53c506abc2cf0c7ffb30de6b4150f1062fac961b504f792e76ddf71 WatchSource:0}: Error finding container 2d7cc86de53c506abc2cf0c7ffb30de6b4150f1062fac961b504f792e76ddf71: Status 404 returned error can't find the container with id 2d7cc86de53c506abc2cf0c7ffb30de6b4150f1062fac961b504f792e76ddf71
Oct 30 17:24:26.421763 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:26.421728    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:24:26.421866 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:26.421799    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:24:26.421866 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:26.421835    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:24:26.474651 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:26.474621    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"]
Oct 30 17:24:26.476594 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:24:26.476567    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd99bf885_403c_47ec_9b74_9c4c57d0e3ef.slice/crio-bd4a48e4088870d7dd6f3d3f9cbe747b70d0c7b1f97de81aaa52360fa7034dc9 WatchSource:0}: Error finding container bd4a48e4088870d7dd6f3d3f9cbe747b70d0c7b1f97de81aaa52360fa7034dc9: Status 404 returned error can't find the container with id bd4a48e4088870d7dd6f3d3f9cbe747b70d0c7b1f97de81aaa52360fa7034dc9
Oct 30 17:24:26.479365 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:26.479335    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:24:26.479460 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:26.479433    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:24:26.479499 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:26.479464    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:24:26.994574 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:24:26.994487    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:24:27.489424 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:27.488365    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894" event={"ID":"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9","Type":"ContainerStarted","Data":"bf8113882c8ca5815be335259b2723f5bc4498ad0c1469616ebfe5b1e7c91ab1"}
Oct 30 17:24:27.489424 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:27.488729    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894" event={"ID":"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9","Type":"ContainerStarted","Data":"2d7cc86de53c506abc2cf0c7ffb30de6b4150f1062fac961b504f792e76ddf71"}
Oct 30 17:24:27.494442 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:27.493349    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh" event={"ID":"d99bf885-403c-47ec-9b74-9c4c57d0e3ef","Type":"ContainerStarted","Data":"c1f033efbd7946641a48ce9aa2746a441c5e2a4acfcf9142135a7aabd8959a02"}
Oct 30 17:24:27.494442 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:27.493438    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh" event={"ID":"d99bf885-403c-47ec-9b74-9c4c57d0e3ef","Type":"ContainerStarted","Data":"bd4a48e4088870d7dd6f3d3f9cbe747b70d0c7b1f97de81aaa52360fa7034dc9"}
Oct 30 17:24:27.562538 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:27.562471    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894" podStartSLOduration=2.562449273 podStartE2EDuration="2.562449273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:27.54711079 +0000 UTC m=+2243.982012996" watchObservedRunningTime="2026-10-30 17:24:27.562449273 +0000 UTC m=+2243.997351478"
Oct 30 17:24:27.635454 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:27.627260    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh" podStartSLOduration=2.627237693 podStartE2EDuration="2.627237693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:27.603443686 +0000 UTC m=+2244.038345891" watchObservedRunningTime="2026-10-30 17:24:27.627237693 +0000 UTC m=+2244.062139899"
Oct 30 17:24:28.100358 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:28.100316    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 17:24:28.100994 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:28.100963    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894" podUID="5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 17:24:28.101434 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:28.101396    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"
Oct 30 17:24:28.136218 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:28.135526    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"
Oct 30 17:24:28.136389 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:28.136283    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 17:24:28.136478 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:28.136386    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh" podUID="d99bf885-403c-47ec-9b74-9c4c57d0e3ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 17:24:29.111059 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:29.111022    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 17:24:29.145095 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:29.141027    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894" podUID="5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 17:24:29.187430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:29.140968    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 17:24:29.212484 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:29.202136    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh" podUID="d99bf885-403c-47ec-9b74-9c4c57d0e3ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 17:24:30.100430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:30.099421    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 17:24:30.100430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:30.099488    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894" podUID="5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 17:24:30.132247 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:30.132202    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 17:24:30.132777 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:30.132734    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh" podUID="d99bf885-403c-47ec-9b74-9c4c57d0e3ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 17:24:31.097431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:31.093818    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 17:24:31.097431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:31.093892    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894" podUID="5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 17:24:31.132775 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:31.132738    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 17:24:31.133252 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:31.132784    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh" podUID="d99bf885-403c-47ec-9b74-9c4c57d0e3ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 17:24:32.105428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:32.101866    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 17:24:32.105428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:32.101953    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894" podUID="5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 17:24:32.132718 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:32.132687    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 17:24:32.132955 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:32.132931    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh" podUID="d99bf885-403c-47ec-9b74-9c4c57d0e3ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 17:24:33.093045 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:33.093007    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 17:24:33.093257 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:33.093084    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894" podUID="5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 17:24:33.132986 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:33.132949    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 17:24:33.133562 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:33.133010    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh" podUID="d99bf885-403c-47ec-9b74-9c4c57d0e3ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 17:24:34.097800 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:34.097762    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"
Oct 30 17:24:34.098336 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:34.098311    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"
Oct 30 17:24:34.100839 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:34.100811    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"
Oct 30 17:24:34.189573 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:34.189535    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"
Oct 30 17:24:34.192822 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:34.192749    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"
Oct 30 17:24:34.196695 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:34.196665    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"
Oct 30 17:24:41.700498 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:41.700456    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"]
Oct 30 17:24:41.701033 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:41.700699    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw" podUID="eead3a6a-f87a-43c0-94fd-e2c0ced708ba" containerName="istio-proxy" containerID="cri-o://259a116c386cf1fc18846376120dcb247ad5a50fd624e12dc12324ead32b6c5f" gracePeriod=30
Oct 30 17:24:46.984928 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:46.984851    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"
Oct 30 17:24:47.075528 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.075494    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-istio-token\" (UniqueName: \"kubernetes.io/projected/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-istio-token\") pod \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") "
Oct 30 17:24:47.075716 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.075603    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-credential-socket\") pod \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") "
Oct 30 17:24:47.075716 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.075630    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-workload-certs\") pod \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") "
Oct 30 17:24:47.075716 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.075681    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-istio-envoy\") pod \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") "
Oct 30 17:24:47.075909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.075743    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-kube-api-access-66crl\" (UniqueName: \"kubernetes.io/projected/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-kube-api-access-66crl\") pod \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") "
Oct 30 17:24:47.075909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.075789    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-workload-socket\") pod \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") "
Oct 30 17:24:47.075909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.075827    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-istio-data\") pod \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") "
Oct 30 17:24:47.075909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.075869    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-istiod-ca-cert\") pod \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") "
Oct 30 17:24:47.076126 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.075920    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-istio-podinfo\") pod \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\" (UID: \"eead3a6a-f87a-43c0-94fd-e2c0ced708ba\") "
Oct 30 17:24:47.076185 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.076120    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-istio-data" pod "eead3a6a-f87a-43c0-94fd-e2c0ced708ba" (UID: "eead3a6a-f87a-43c0-94fd-e2c0ced708ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:47.076341 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.076323    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-workload-socket" pod "eead3a6a-f87a-43c0-94fd-e2c0ced708ba" (UID: "eead3a6a-f87a-43c0-94fd-e2c0ced708ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:47.076630 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.076611    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:24:47.076725 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.076634    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:24:47.076725 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.076693    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-credential-socket" pod "eead3a6a-f87a-43c0-94fd-e2c0ced708ba" (UID: "eead3a6a-f87a-43c0-94fd-e2c0ced708ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:47.076804 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.076723    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-workload-certs" pod "eead3a6a-f87a-43c0-94fd-e2c0ced708ba" (UID: "eead3a6a-f87a-43c0-94fd-e2c0ced708ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:47.076804 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.076737    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-istiod-ca-cert" pod "eead3a6a-f87a-43c0-94fd-e2c0ced708ba" (UID: "eead3a6a-f87a-43c0-94fd-e2c0ced708ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:24:47.078244 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.078198    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-istio-token" pod "eead3a6a-f87a-43c0-94fd-e2c0ced708ba" (UID: "eead3a6a-f87a-43c0-94fd-e2c0ced708ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:24:47.078739 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.078715    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-istio-envoy" pod "eead3a6a-f87a-43c0-94fd-e2c0ced708ba" (UID: "eead3a6a-f87a-43c0-94fd-e2c0ced708ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:47.078833 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.078776    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-kube-api-access-66crl" pod "eead3a6a-f87a-43c0-94fd-e2c0ced708ba" (UID: "eead3a6a-f87a-43c0-94fd-e2c0ced708ba"). InnerVolumeSpecName "kube-api-access-66crl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:24:47.078875 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.078841    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-istio-podinfo" pod "eead3a6a-f87a-43c0-94fd-e2c0ced708ba" (UID: "eead3a6a-f87a-43c0-94fd-e2c0ced708ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:24:47.178226 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.178184    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:24:47.178226 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.178226    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:24:47.178528 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.178242    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:24:47.178528 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.178255    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:24:47.178528 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.178269    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66crl\" (UniqueName: \"kubernetes.io/projected/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-kube-api-access-66crl\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:24:47.178528 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.178283    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:24:47.178528 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.178297    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eead3a6a-f87a-43c0-94fd-e2c0ced708ba-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:24:47.651022 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.650983    4433 generic.go:356] "Generic (PLEG): container finished" podID="eead3a6a-f87a-43c0-94fd-e2c0ced708ba" containerID="259a116c386cf1fc18846376120dcb247ad5a50fd624e12dc12324ead32b6c5f" exitCode=0
Oct 30 17:24:47.651232 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.651034    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"
Oct 30 17:24:47.651232 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.651079    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw" event={"ID":"eead3a6a-f87a-43c0-94fd-e2c0ced708ba","Type":"ContainerDied","Data":"259a116c386cf1fc18846376120dcb247ad5a50fd624e12dc12324ead32b6c5f"}
Oct 30 17:24:47.651232 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.651141    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw" event={"ID":"eead3a6a-f87a-43c0-94fd-e2c0ced708ba","Type":"ContainerDied","Data":"5e03f5fb9f3f612866f7a8117a13b4a43df4c3c0982da5a37345aa0d33870680"}
Oct 30 17:24:47.651232 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.651178    4433 scope.go:122] "RemoveContainer" containerID="259a116c386cf1fc18846376120dcb247ad5a50fd624e12dc12324ead32b6c5f"
Oct 30 17:24:47.663498 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.663474    4433 scope.go:122] "RemoveContainer" containerID="259a116c386cf1fc18846376120dcb247ad5a50fd624e12dc12324ead32b6c5f"
Oct 30 17:24:47.663834 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:24:47.663808    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"259a116c386cf1fc18846376120dcb247ad5a50fd624e12dc12324ead32b6c5f\": container with ID starting with 259a116c386cf1fc18846376120dcb247ad5a50fd624e12dc12324ead32b6c5f not found: ID does not exist" containerID="259a116c386cf1fc18846376120dcb247ad5a50fd624e12dc12324ead32b6c5f"
Oct 30 17:24:47.663942 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.663847    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"259a116c386cf1fc18846376120dcb247ad5a50fd624e12dc12324ead32b6c5f"} err="failed to get container status \"259a116c386cf1fc18846376120dcb247ad5a50fd624e12dc12324ead32b6c5f\": rpc error: code = NotFound desc = could not find container \"259a116c386cf1fc18846376120dcb247ad5a50fd624e12dc12324ead32b6c5f\": container with ID starting with 259a116c386cf1fc18846376120dcb247ad5a50fd624e12dc12324ead32b6c5f not found: ID does not exist"
Oct 30 17:24:47.716203 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.716165    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"]
Oct 30 17:24:47.727538 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:47.727506    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-wbhhw"]
Oct 30 17:24:49.695865 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:49.695824    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eead3a6a-f87a-43c0-94fd-e2c0ced708ba" path="/var/lib/kubelet/pods/eead3a6a-f87a-43c0-94fd-e2c0ced708ba/volumes"
Oct 30 17:24:56.184591 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:24:56.184555    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:24:56.979921 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:24:56.979885    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:25:03.915726 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:03.915687    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh"]
Oct 30 17:25:14.646572 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:14.646530    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n"]
Oct 30 17:25:18.937019 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:18.936930    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh" podUID="9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f" containerName="main" containerID="cri-o://eda6c7e1c95c50c8ff4d91944235528eef01c7af507b789845bb20b4612a37e4" gracePeriod=45
Oct 30 17:25:18.957259 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:18.957218    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll"]
Oct 30 17:25:18.957797 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:18.957753    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll" podUID="95172c31-1b94-47fd-b95e-8170b67562ca" containerName="maas-api" containerID="cri-o://173af07945d3b74f12e8981b620b10898c1f9e284ff0273d920d20caf4fd8d46" gracePeriod=30
Oct 30 17:25:19.050845 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.050787    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh" podUID="9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.133:8000/ready\": dial tcp 10.133.0.133:8000: connect: connection refused"
Oct 30 17:25:19.415118 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.415091    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh"
Oct 30 17:25:19.418919 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.418895    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll"
Oct 30 17:25:19.500800 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.500711    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-dshm\") pod \"9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f\" (UID: \"9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f\") "
Oct 30 17:25:19.500982 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.500808    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95172c31-1b94-47fd-b95e-8170b67562ca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/95172c31-1b94-47fd-b95e-8170b67562ca-metrics-certs\") pod \"95172c31-1b94-47fd-b95e-8170b67562ca\" (UID: \"95172c31-1b94-47fd-b95e-8170b67562ca\") "
Oct 30 17:25:19.500982 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.500851    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95172c31-1b94-47fd-b95e-8170b67562ca-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95172c31-1b94-47fd-b95e-8170b67562ca-openshift-service-ca\") pod \"95172c31-1b94-47fd-b95e-8170b67562ca\" (UID: \"95172c31-1b94-47fd-b95e-8170b67562ca\") "
Oct 30 17:25:19.500982 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.500876    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95172c31-1b94-47fd-b95e-8170b67562ca-kube-api-access-dfkwh\" (UniqueName: \"kubernetes.io/projected/95172c31-1b94-47fd-b95e-8170b67562ca-kube-api-access-dfkwh\") pod \"95172c31-1b94-47fd-b95e-8170b67562ca\" (UID: \"95172c31-1b94-47fd-b95e-8170b67562ca\") "
Oct 30 17:25:19.500982 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.500947    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-tls-certs\" (UniqueName: \"kubernetes.io/secret/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-tls-certs\") pod \"9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f\" (UID: \"9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f\") "
Oct 30 17:25:19.501202 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.501044    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-model-cache\") pod \"9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f\" (UID: \"9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f\") "
Oct 30 17:25:19.501202 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.501147    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-tmp-dir\") pod \"9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f\" (UID: \"9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f\") "
Oct 30 17:25:19.501202 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.501187    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-home\" (UniqueName: \"kubernetes.io/empty-dir/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-home\") pod \"9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f\" (UID: \"9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f\") "
Oct 30 17:25:19.501353 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.501258    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95172c31-1b94-47fd-b95e-8170b67562ca-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95172c31-1b94-47fd-b95e-8170b67562ca-maas-api-tls\") pod \"95172c31-1b94-47fd-b95e-8170b67562ca\" (UID: \"95172c31-1b94-47fd-b95e-8170b67562ca\") "
Oct 30 17:25:19.501353 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.501298    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-kube-api-access-f79kf\" (UniqueName: \"kubernetes.io/projected/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-kube-api-access-f79kf\") pod \"9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f\" (UID: \"9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f\") "
Oct 30 17:25:19.503842 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.503799    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-dshm" pod "9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f" (UID: "9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:25:19.504272 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.504236    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95172c31-1b94-47fd-b95e-8170b67562ca-openshift-service-ca" pod "95172c31-1b94-47fd-b95e-8170b67562ca" (UID: "95172c31-1b94-47fd-b95e-8170b67562ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:25:19.504570 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.504545    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-model-cache" pod "9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f" (UID: "9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:25:19.507757 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.504737    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-home" pod "9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f" (UID: "9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:25:19.508242 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.508206    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95172c31-1b94-47fd-b95e-8170b67562ca-maas-api-tls" pod "95172c31-1b94-47fd-b95e-8170b67562ca" (UID: "95172c31-1b94-47fd-b95e-8170b67562ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:25:19.509186 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.509148    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-tmp-dir" pod "9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f" (UID: "9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:25:19.520290 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.514727    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95172c31-1b94-47fd-b95e-8170b67562ca-metrics-certs" pod "95172c31-1b94-47fd-b95e-8170b67562ca" (UID: "95172c31-1b94-47fd-b95e-8170b67562ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:25:19.520290 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.514742    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-tls-certs" pod "9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f" (UID: "9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:25:19.520290 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.514889    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-kube-api-access-f79kf" pod "9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f" (UID: "9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f"). InnerVolumeSpecName "kube-api-access-f79kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:25:19.535326 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.534108    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95172c31-1b94-47fd-b95e-8170b67562ca-kube-api-access-dfkwh" pod "95172c31-1b94-47fd-b95e-8170b67562ca" (UID: "95172c31-1b94-47fd-b95e-8170b67562ca"). InnerVolumeSpecName "kube-api-access-dfkwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:25:19.610028 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.609908    4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95172c31-1b94-47fd-b95e-8170b67562ca-metrics-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:25:19.610028 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.609952    4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95172c31-1b94-47fd-b95e-8170b67562ca-openshift-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:25:19.610028 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.609969    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfkwh\" (UniqueName: \"kubernetes.io/projected/95172c31-1b94-47fd-b95e-8170b67562ca-kube-api-access-dfkwh\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:25:19.610028 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.609983    4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-tls-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:25:19.610028 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.609998    4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-model-cache\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:25:19.610028 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.610010    4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-tmp-dir\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:25:19.610028 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.610021    4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-home\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:25:19.610028 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.610033    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95172c31-1b94-47fd-b95e-8170b67562ca-maas-api-tls\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:25:19.610028 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.610045    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f79kf\" (UniqueName: \"kubernetes.io/projected/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-kube-api-access-f79kf\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:25:19.611745 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.610057    4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f-dshm\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:25:19.923246 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.923160    4433 generic.go:356] "Generic (PLEG): container finished" podID="9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f" containerID="eda6c7e1c95c50c8ff4d91944235528eef01c7af507b789845bb20b4612a37e4" exitCode=0
Oct 30 17:25:19.923246 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.923209    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh"
Oct 30 17:25:19.923476 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.923250    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh" event={"ID":"9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f","Type":"ContainerDied","Data":"eda6c7e1c95c50c8ff4d91944235528eef01c7af507b789845bb20b4612a37e4"}
Oct 30 17:25:19.923476 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.923291    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh" event={"ID":"9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f","Type":"ContainerDied","Data":"2f8603eeb29eaeab66c4eb146957d1655367fddb2936e75178bd7b1a98238223"}
Oct 30 17:25:19.923476 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.923313    4433 scope.go:122] "RemoveContainer" containerID="eda6c7e1c95c50c8ff4d91944235528eef01c7af507b789845bb20b4612a37e4"
Oct 30 17:25:19.924788 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.924768    4433 generic.go:356] "Generic (PLEG): container finished" podID="95172c31-1b94-47fd-b95e-8170b67562ca" containerID="173af07945d3b74f12e8981b620b10898c1f9e284ff0273d920d20caf4fd8d46" exitCode=0
Oct 30 17:25:19.924886 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.924807    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll" event={"ID":"95172c31-1b94-47fd-b95e-8170b67562ca","Type":"ContainerDied","Data":"173af07945d3b74f12e8981b620b10898c1f9e284ff0273d920d20caf4fd8d46"}
Oct 30 17:25:19.924886 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.924817    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll"
Oct 30 17:25:19.924886 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.924847    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll" event={"ID":"95172c31-1b94-47fd-b95e-8170b67562ca","Type":"ContainerDied","Data":"db56297eedf36c22f69c3bd6a1fdb08ced8e421b61624033e08b928d12996432"}
Oct 30 17:25:19.934608 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.934580    4433 scope.go:122] "RemoveContainer" containerID="eda6c7e1c95c50c8ff4d91944235528eef01c7af507b789845bb20b4612a37e4"
Oct 30 17:25:19.935031 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:25:19.935002    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eda6c7e1c95c50c8ff4d91944235528eef01c7af507b789845bb20b4612a37e4\": container with ID starting with eda6c7e1c95c50c8ff4d91944235528eef01c7af507b789845bb20b4612a37e4 not found: ID does not exist" containerID="eda6c7e1c95c50c8ff4d91944235528eef01c7af507b789845bb20b4612a37e4"
Oct 30 17:25:19.935158 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.935041    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eda6c7e1c95c50c8ff4d91944235528eef01c7af507b789845bb20b4612a37e4"} err="failed to get container status \"eda6c7e1c95c50c8ff4d91944235528eef01c7af507b789845bb20b4612a37e4\": rpc error: code = NotFound desc = could not find container \"eda6c7e1c95c50c8ff4d91944235528eef01c7af507b789845bb20b4612a37e4\": container with ID starting with eda6c7e1c95c50c8ff4d91944235528eef01c7af507b789845bb20b4612a37e4 not found: ID does not exist"
Oct 30 17:25:19.935158 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.935070    4433 scope.go:122] "RemoveContainer" containerID="173af07945d3b74f12e8981b620b10898c1f9e284ff0273d920d20caf4fd8d46"
Oct 30 17:25:19.947540 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.947326    4433 scope.go:122] "RemoveContainer" containerID="173af07945d3b74f12e8981b620b10898c1f9e284ff0273d920d20caf4fd8d46"
Oct 30 17:25:19.947870 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:25:19.947713    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"173af07945d3b74f12e8981b620b10898c1f9e284ff0273d920d20caf4fd8d46\": container with ID starting with 173af07945d3b74f12e8981b620b10898c1f9e284ff0273d920d20caf4fd8d46 not found: ID does not exist" containerID="173af07945d3b74f12e8981b620b10898c1f9e284ff0273d920d20caf4fd8d46"
Oct 30 17:25:19.947870 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.947749    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"173af07945d3b74f12e8981b620b10898c1f9e284ff0273d920d20caf4fd8d46"} err="failed to get container status \"173af07945d3b74f12e8981b620b10898c1f9e284ff0273d920d20caf4fd8d46\": rpc error: code = NotFound desc = could not find container \"173af07945d3b74f12e8981b620b10898c1f9e284ff0273d920d20caf4fd8d46\": container with ID starting with 173af07945d3b74f12e8981b620b10898c1f9e284ff0273d920d20caf4fd8d46 not found: ID does not exist"
Oct 30 17:25:19.978723 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.978684    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh"]
Oct 30 17:25:19.988387 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:19.988354    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c45877/test-model-c45877-kserve-7bfd8f46f5-zx7fh"]
Oct 30 17:25:20.018977 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:20.018940    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll"]
Oct 30 17:25:20.023019 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:20.022987    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e5ac8f-855b6457bb-p24ll"]
Oct 30 17:25:21.695583 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:21.695547    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95172c31-1b94-47fd-b95e-8170b67562ca" path="/var/lib/kubelet/pods/95172c31-1b94-47fd-b95e-8170b67562ca/volumes"
Oct 30 17:25:21.696069 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:21.696054    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f" path="/var/lib/kubelet/pods/9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f/volumes"
Oct 30 17:25:21.732351 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:21.732315    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-76f8ff444c-gtjjg"]
Oct 30 17:25:21.732914 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:21.732899    4433 cpu_manager.go:414] "containerMap: removing container" podUID="9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f" containerName="main"
Oct 30 17:25:21.732959 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:21.732915    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f" containerName="main"
Oct 30 17:25:21.732959 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:21.732955    4433 cpu_manager.go:414] "containerMap: removing container" podUID="eead3a6a-f87a-43c0-94fd-e2c0ced708ba" containerName="istio-proxy"
Oct 30 17:25:21.733055 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:21.732961    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eead3a6a-f87a-43c0-94fd-e2c0ced708ba" containerName="istio-proxy"
Oct 30 17:25:21.733055 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:21.732975    4433 cpu_manager.go:414] "containerMap: removing container" podUID="95172c31-1b94-47fd-b95e-8170b67562ca" containerName="maas-api"
Oct 30 17:25:21.733055 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:21.732980    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95172c31-1b94-47fd-b95e-8170b67562ca" containerName="maas-api"
Oct 30 17:25:21.733141 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:21.733057    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="eead3a6a-f87a-43c0-94fd-e2c0ced708ba" containerName="istio-proxy"
Oct 30 17:25:21.733141 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:21.733065    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="95172c31-1b94-47fd-b95e-8170b67562ca" containerName="maas-api"
Oct 30 17:25:21.733141 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:21.733078    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fc5e447-ac41-4ca7-a0d2-35a06e5dd46f" containerName="main"
Oct 30 17:25:21.737758 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:21.737736    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-76f8ff444c-gtjjg"
Oct 30 17:25:21.756286 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:21.756256    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-76f8ff444c-gtjjg"]
Oct 30 17:25:21.764494 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:21.764463    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-abe243-dockercfg-ztpcd\""
Oct 30 17:25:21.764715 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:21.764489    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-abe243\""
Oct 30 17:25:21.956990 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:21.956877    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d2fff10-15e7-48da-abfd-b3897f0ccba4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-abe243-76f8ff444c-gtjjg\" (UID: \"3d2fff10-15e7-48da-abfd-b3897f0ccba4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-76f8ff444c-gtjjg"
Oct 30 17:25:21.961653 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:21.961597    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svq9j\" (UniqueName: \"kubernetes.io/projected/3d2fff10-15e7-48da-abfd-b3897f0ccba4-kube-api-access-svq9j\") pod \"payload-pre-processing-e2e-ipp-a-abe243-76f8ff444c-gtjjg\" (UID: \"3d2fff10-15e7-48da-abfd-b3897f0ccba4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-76f8ff444c-gtjjg"
Oct 30 17:25:22.063534 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:22.063495    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d2fff10-15e7-48da-abfd-b3897f0ccba4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-abe243-76f8ff444c-gtjjg\" (UID: \"3d2fff10-15e7-48da-abfd-b3897f0ccba4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-76f8ff444c-gtjjg"
Oct 30 17:25:22.063722 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:22.063639    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svq9j\" (UniqueName: \"kubernetes.io/projected/3d2fff10-15e7-48da-abfd-b3897f0ccba4-kube-api-access-svq9j\") pod \"payload-pre-processing-e2e-ipp-a-abe243-76f8ff444c-gtjjg\" (UID: \"3d2fff10-15e7-48da-abfd-b3897f0ccba4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-76f8ff444c-gtjjg"
Oct 30 17:25:22.064347 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:22.064324    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d2fff10-15e7-48da-abfd-b3897f0ccba4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-abe243-76f8ff444c-gtjjg\" (UID: \"3d2fff10-15e7-48da-abfd-b3897f0ccba4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-76f8ff444c-gtjjg"
Oct 30 17:25:22.086677 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:22.086643    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svq9j\" (UniqueName: \"kubernetes.io/projected/3d2fff10-15e7-48da-abfd-b3897f0ccba4-kube-api-access-svq9j\") pod \"payload-pre-processing-e2e-ipp-a-abe243-76f8ff444c-gtjjg\" (UID: \"3d2fff10-15e7-48da-abfd-b3897f0ccba4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-76f8ff444c-gtjjg"
Oct 30 17:25:22.352187 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:22.352104    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-76f8ff444c-gtjjg"
Oct 30 17:25:22.655609 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:22.655579    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-76f8ff444c-gtjjg"]
Oct 30 17:25:22.658202 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:25:22.658170    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d2fff10_15e7_48da_abfd_b3897f0ccba4.slice/crio-38afd138bcd8cfd1c60e23683af188ca35ebb2d5b9ef1f5fe0aff959c89a44b4 WatchSource:0}: Error finding container 38afd138bcd8cfd1c60e23683af188ca35ebb2d5b9ef1f5fe0aff959c89a44b4: Status 404 returned error can't find the container with id 38afd138bcd8cfd1c60e23683af188ca35ebb2d5b9ef1f5fe0aff959c89a44b4
Oct 30 17:25:22.975685 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:22.975642    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-76f8ff444c-gtjjg" event={"ID":"3d2fff10-15e7-48da-abfd-b3897f0ccba4","Type":"ContainerStarted","Data":"88e5fb7abcfbcfdc599be86cf172d396c1d064191c8bb9dc2ef132945eb4e600"}
Oct 30 17:25:22.976092 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:22.975697    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-76f8ff444c-gtjjg" event={"ID":"3d2fff10-15e7-48da-abfd-b3897f0ccba4","Type":"ContainerStarted","Data":"38afd138bcd8cfd1c60e23683af188ca35ebb2d5b9ef1f5fe0aff959c89a44b4"}
Oct 30 17:25:22.976092 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:22.975721    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-76f8ff444c-gtjjg"
Oct 30 17:25:23.017652 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:23.017599    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-76f8ff444c-gtjjg" podStartSLOduration=2.017584713 podStartE2EDuration="2.017584713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:25:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:25:23.015570646 +0000 UTC m=+2299.450472848" watchObservedRunningTime="2026-10-30 17:25:23.017584713 +0000 UTC m=+2299.452486913"
Oct 30 17:25:26.978297 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:25:26.978259    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:25:29.097091 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.097043    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"]
Oct 30 17:25:29.103460 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.102520    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:25:29.120565 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.120533    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-469ad7-openshift-default-dockercfg-q2qkg\""
Oct 30 17:25:29.139130 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.135681    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"]
Oct 30 17:25:29.175181 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.173675    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d74c8b0b-844e-493f-a994-5b83b7b73589-istio-data\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:25:29.175181 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.173730    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrgbt\" (UniqueName: \"kubernetes.io/projected/d74c8b0b-844e-493f-a994-5b83b7b73589-kube-api-access-mrgbt\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:25:29.175181 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.173765    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d74c8b0b-844e-493f-a994-5b83b7b73589-istio-token\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:25:29.175181 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.173808    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d74c8b0b-844e-493f-a994-5b83b7b73589-istio-podinfo\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:25:29.175181 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.173917    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d74c8b0b-844e-493f-a994-5b83b7b73589-istio-envoy\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:25:29.175181 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.174019    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d74c8b0b-844e-493f-a994-5b83b7b73589-credential-socket\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:25:29.175181 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.174056    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d74c8b0b-844e-493f-a994-5b83b7b73589-istiod-ca-cert\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:25:29.175181 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.174085    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d74c8b0b-844e-493f-a994-5b83b7b73589-workload-socket\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:25:29.175181 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.174124    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d74c8b0b-844e-493f-a994-5b83b7b73589-workload-certs\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:25:29.275084 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.275047    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d74c8b0b-844e-493f-a994-5b83b7b73589-istio-envoy\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:25:29.275313 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.275151    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d74c8b0b-844e-493f-a994-5b83b7b73589-credential-socket\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:25:29.275313 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.275177    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d74c8b0b-844e-493f-a994-5b83b7b73589-istiod-ca-cert\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:25:29.275313 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.275200    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d74c8b0b-844e-493f-a994-5b83b7b73589-workload-socket\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:25:29.275313 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.275239    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d74c8b0b-844e-493f-a994-5b83b7b73589-workload-certs\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:25:29.275575 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.275327    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d74c8b0b-844e-493f-a994-5b83b7b73589-istio-data\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:25:29.275575 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.275352    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrgbt\" (UniqueName: \"kubernetes.io/projected/d74c8b0b-844e-493f-a994-5b83b7b73589-kube-api-access-mrgbt\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:25:29.275575 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.275385    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d74c8b0b-844e-493f-a994-5b83b7b73589-istio-token\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:25:29.275575 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.275454    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d74c8b0b-844e-493f-a994-5b83b7b73589-istio-podinfo\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:25:29.275789 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.275668    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d74c8b0b-844e-493f-a994-5b83b7b73589-credential-socket\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:25:29.275980 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.275957    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d74c8b0b-844e-493f-a994-5b83b7b73589-istio-data\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:25:29.276093 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.276067    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d74c8b0b-844e-493f-a994-5b83b7b73589-istiod-ca-cert\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:25:29.276264 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.276241    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d74c8b0b-844e-493f-a994-5b83b7b73589-workload-socket\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:25:29.276368 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.276345    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d74c8b0b-844e-493f-a994-5b83b7b73589-workload-certs\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:25:29.279844 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.279704    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d74c8b0b-844e-493f-a994-5b83b7b73589-istio-envoy\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:25:29.281030 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.280187    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d74c8b0b-844e-493f-a994-5b83b7b73589-istio-podinfo\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:25:29.320426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.314726    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d74c8b0b-844e-493f-a994-5b83b7b73589-istio-token\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:25:29.320426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.318714    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrgbt\" (UniqueName: \"kubernetes.io/projected/d74c8b0b-844e-493f-a994-5b83b7b73589-kube-api-access-mrgbt\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:25:29.425484 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.422106    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:25:29.675372 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.675329    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n" podUID="1ca3e5af-f7e2-47d5-bc76-66d298b16971" containerName="main" containerID="cri-o://093262a71277c7703fe3168c5cdb17d6183bef9017575a7319cfb4bc8a72ae87" gracePeriod=45
Oct 30 17:25:29.777622 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.777564    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n" podUID="1ca3e5af-f7e2-47d5-bc76-66d298b16971" containerName="main" probeResult="failure" output="Get \"https://10.133.0.136:8000/ready\": dial tcp 10.133.0.136:8000: connect: connection refused"
Oct 30 17:25:29.831641 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.831602    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"]
Oct 30 17:25:29.836282 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:25:29.836248    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd74c8b0b_844e_493f_a994_5b83b7b73589.slice/crio-df0a4718e9be52fd56fd1338d028edb1205652744b1f126c58fe73655fe59109 WatchSource:0}: Error finding container df0a4718e9be52fd56fd1338d028edb1205652744b1f126c58fe73655fe59109: Status 404 returned error can't find the container with id df0a4718e9be52fd56fd1338d028edb1205652744b1f126c58fe73655fe59109
Oct 30 17:25:29.839058 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.839021    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:25:29.839197 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.839096    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:25:29.839197 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.839134    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:25:29.953491 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:29.953460    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n"
Oct 30 17:25:30.067137 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:30.067106    4433 generic.go:356] "Generic (PLEG): container finished" podID="1ca3e5af-f7e2-47d5-bc76-66d298b16971" containerID="093262a71277c7703fe3168c5cdb17d6183bef9017575a7319cfb4bc8a72ae87" exitCode=0
Oct 30 17:25:30.067321 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:30.067152    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n"
Oct 30 17:25:30.067321 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:30.067194    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n" event={"ID":"1ca3e5af-f7e2-47d5-bc76-66d298b16971","Type":"ContainerDied","Data":"093262a71277c7703fe3168c5cdb17d6183bef9017575a7319cfb4bc8a72ae87"}
Oct 30 17:25:30.067321 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:30.067236    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n" event={"ID":"1ca3e5af-f7e2-47d5-bc76-66d298b16971","Type":"ContainerDied","Data":"9f8fdf8bbcc21cbf7407c8a3bfd15d71308660962e6d07da10270e3fa177ebb1"}
Oct 30 17:25:30.067321 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:30.067259    4433 scope.go:122] "RemoveContainer" containerID="093262a71277c7703fe3168c5cdb17d6183bef9017575a7319cfb4bc8a72ae87"
Oct 30 17:25:30.069283 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:30.069258    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww" event={"ID":"d74c8b0b-844e-493f-a994-5b83b7b73589","Type":"ContainerStarted","Data":"890ea6dad27d4f3b9958d2f10ba005dad229051350aa4f19a895e1390d0d8954"}
Oct 30 17:25:30.069657 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:30.069313    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww" event={"ID":"d74c8b0b-844e-493f-a994-5b83b7b73589","Type":"ContainerStarted","Data":"df0a4718e9be52fd56fd1338d028edb1205652744b1f126c58fe73655fe59109"}
Oct 30 17:25:30.078774 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:30.078749    4433 scope.go:122] "RemoveContainer" containerID="093262a71277c7703fe3168c5cdb17d6183bef9017575a7319cfb4bc8a72ae87"
Oct 30 17:25:30.079137 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:25:30.079107    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"093262a71277c7703fe3168c5cdb17d6183bef9017575a7319cfb4bc8a72ae87\": container with ID starting with 093262a71277c7703fe3168c5cdb17d6183bef9017575a7319cfb4bc8a72ae87 not found: ID does not exist" containerID="093262a71277c7703fe3168c5cdb17d6183bef9017575a7319cfb4bc8a72ae87"
Oct 30 17:25:30.079201 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:30.079154    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"093262a71277c7703fe3168c5cdb17d6183bef9017575a7319cfb4bc8a72ae87"} err="failed to get container status \"093262a71277c7703fe3168c5cdb17d6183bef9017575a7319cfb4bc8a72ae87\": rpc error: code = NotFound desc = could not find container \"093262a71277c7703fe3168c5cdb17d6183bef9017575a7319cfb4bc8a72ae87\": container with ID starting with 093262a71277c7703fe3168c5cdb17d6183bef9017575a7319cfb4bc8a72ae87 not found: ID does not exist"
Oct 30 17:25:30.088678 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:30.088634    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ca3e5af-f7e2-47d5-bc76-66d298b16971-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ca3e5af-f7e2-47d5-bc76-66d298b16971-model-cache\") pod \"1ca3e5af-f7e2-47d5-bc76-66d298b16971\" (UID: \"1ca3e5af-f7e2-47d5-bc76-66d298b16971\") "
Oct 30 17:25:30.088856 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:30.088686    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ca3e5af-f7e2-47d5-bc76-66d298b16971-kube-api-access-fqpgb\" (UniqueName: \"kubernetes.io/projected/1ca3e5af-f7e2-47d5-bc76-66d298b16971-kube-api-access-fqpgb\") pod \"1ca3e5af-f7e2-47d5-bc76-66d298b16971\" (UID: \"1ca3e5af-f7e2-47d5-bc76-66d298b16971\") "
Oct 30 17:25:30.088856 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:30.088727    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ca3e5af-f7e2-47d5-bc76-66d298b16971-home\" (UniqueName: \"kubernetes.io/empty-dir/1ca3e5af-f7e2-47d5-bc76-66d298b16971-home\") pod \"1ca3e5af-f7e2-47d5-bc76-66d298b16971\" (UID: \"1ca3e5af-f7e2-47d5-bc76-66d298b16971\") "
Oct 30 17:25:30.088856 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:30.088766    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ca3e5af-f7e2-47d5-bc76-66d298b16971-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ca3e5af-f7e2-47d5-bc76-66d298b16971-tmp-dir\") pod \"1ca3e5af-f7e2-47d5-bc76-66d298b16971\" (UID: \"1ca3e5af-f7e2-47d5-bc76-66d298b16971\") "
Oct 30 17:25:30.089003 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:30.088867    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ca3e5af-f7e2-47d5-bc76-66d298b16971-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ca3e5af-f7e2-47d5-bc76-66d298b16971-dshm\") pod \"1ca3e5af-f7e2-47d5-bc76-66d298b16971\" (UID: \"1ca3e5af-f7e2-47d5-bc76-66d298b16971\") "
Oct 30 17:25:30.089003 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:30.088918    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ca3e5af-f7e2-47d5-bc76-66d298b16971-tls-certs\" (UniqueName: \"kubernetes.io/secret/1ca3e5af-f7e2-47d5-bc76-66d298b16971-tls-certs\") pod \"1ca3e5af-f7e2-47d5-bc76-66d298b16971\" (UID: \"1ca3e5af-f7e2-47d5-bc76-66d298b16971\") "
Oct 30 17:25:30.089124 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:30.089104    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca3e5af-f7e2-47d5-bc76-66d298b16971-home" pod "1ca3e5af-f7e2-47d5-bc76-66d298b16971" (UID: "1ca3e5af-f7e2-47d5-bc76-66d298b16971"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:25:30.089186 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:30.089161    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca3e5af-f7e2-47d5-bc76-66d298b16971-model-cache" pod "1ca3e5af-f7e2-47d5-bc76-66d298b16971" (UID: "1ca3e5af-f7e2-47d5-bc76-66d298b16971"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:25:30.089690 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:30.089481    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca3e5af-f7e2-47d5-bc76-66d298b16971-tmp-dir" pod "1ca3e5af-f7e2-47d5-bc76-66d298b16971" (UID: "1ca3e5af-f7e2-47d5-bc76-66d298b16971"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:25:30.089690 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:30.089677    4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ca3e5af-f7e2-47d5-bc76-66d298b16971-model-cache\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:25:30.089940 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:30.089699    4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ca3e5af-f7e2-47d5-bc76-66d298b16971-home\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:25:30.089940 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:30.089711    4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ca3e5af-f7e2-47d5-bc76-66d298b16971-tmp-dir\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:25:30.091667 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:30.091597    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ca3e5af-f7e2-47d5-bc76-66d298b16971-tls-certs" pod "1ca3e5af-f7e2-47d5-bc76-66d298b16971" (UID: "1ca3e5af-f7e2-47d5-bc76-66d298b16971"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:25:30.091770 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:30.091717    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca3e5af-f7e2-47d5-bc76-66d298b16971-kube-api-access-fqpgb" pod "1ca3e5af-f7e2-47d5-bc76-66d298b16971" (UID: "1ca3e5af-f7e2-47d5-bc76-66d298b16971"). InnerVolumeSpecName "kube-api-access-fqpgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:25:30.091987 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:30.091967    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca3e5af-f7e2-47d5-bc76-66d298b16971-dshm" pod "1ca3e5af-f7e2-47d5-bc76-66d298b16971" (UID: "1ca3e5af-f7e2-47d5-bc76-66d298b16971"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:25:30.129459 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:30.127978    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww" podStartSLOduration=1.127961255 podStartE2EDuration="1.127961255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:25:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:25:30.125922423 +0000 UTC m=+2306.560824655" watchObservedRunningTime="2026-10-30 17:25:30.127961255 +0000 UTC m=+2306.562863463"
Oct 30 17:25:30.191157 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:30.191054    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqpgb\" (UniqueName: \"kubernetes.io/projected/1ca3e5af-f7e2-47d5-bc76-66d298b16971-kube-api-access-fqpgb\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:25:30.191157 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:30.191089    4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ca3e5af-f7e2-47d5-bc76-66d298b16971-dshm\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:25:30.191157 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:30.191103    4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ca3e5af-f7e2-47d5-bc76-66d298b16971-tls-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:25:30.414569 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:30.414530    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n"]
Oct 30 17:25:30.419109 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:30.419077    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-001a96/auth-test-model-001a96-kserve-86688f9d4f-8dh2n"]
Oct 30 17:25:30.423034 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:30.423009    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:25:30.429441 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:30.429389    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:25:31.078028 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:31.077998    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:25:31.079047 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:31.079022    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:25:31.697569 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:31.697534    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ca3e5af-f7e2-47d5-bc76-66d298b16971" path="/var/lib/kubelet/pods/1ca3e5af-f7e2-47d5-bc76-66d298b16971/volumes"
Oct 30 17:25:33.986465 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:33.986432    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-76f8ff444c-gtjjg"
Oct 30 17:25:42.949908 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:42.949872    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr"]
Oct 30 17:25:42.950489 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:42.950135    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr" podUID="d47a5a77-8d4f-401b-b4a3-80c2d27718a9" containerName="maas-api" containerID="cri-o://54774d1a67235481922c6a7195c597c91ec65844830283a4bf49c8eb5a6ca5b9" gracePeriod=30
Oct 30 17:25:43.158583 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:43.158545    4433 generic.go:356] "Generic (PLEG): container finished" podID="d47a5a77-8d4f-401b-b4a3-80c2d27718a9" containerID="54774d1a67235481922c6a7195c597c91ec65844830283a4bf49c8eb5a6ca5b9" exitCode=0
Oct 30 17:25:43.158756 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:43.158632    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr" event={"ID":"d47a5a77-8d4f-401b-b4a3-80c2d27718a9","Type":"ContainerDied","Data":"54774d1a67235481922c6a7195c597c91ec65844830283a4bf49c8eb5a6ca5b9"}
Oct 30 17:25:43.396428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:43.388960    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr"
Oct 30 17:25:43.524721 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:43.524685    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d47a5a77-8d4f-401b-b4a3-80c2d27718a9-kube-api-access-zt744\" (UniqueName: \"kubernetes.io/projected/d47a5a77-8d4f-401b-b4a3-80c2d27718a9-kube-api-access-zt744\") pod \"d47a5a77-8d4f-401b-b4a3-80c2d27718a9\" (UID: \"d47a5a77-8d4f-401b-b4a3-80c2d27718a9\") "
Oct 30 17:25:43.524935 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:43.524764    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d47a5a77-8d4f-401b-b4a3-80c2d27718a9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d47a5a77-8d4f-401b-b4a3-80c2d27718a9-maas-api-tls\") pod \"d47a5a77-8d4f-401b-b4a3-80c2d27718a9\" (UID: \"d47a5a77-8d4f-401b-b4a3-80c2d27718a9\") "
Oct 30 17:25:43.524935 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:43.524869    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d47a5a77-8d4f-401b-b4a3-80c2d27718a9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d47a5a77-8d4f-401b-b4a3-80c2d27718a9-metrics-certs\") pod \"d47a5a77-8d4f-401b-b4a3-80c2d27718a9\" (UID: \"d47a5a77-8d4f-401b-b4a3-80c2d27718a9\") "
Oct 30 17:25:43.525071 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:43.525019    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d47a5a77-8d4f-401b-b4a3-80c2d27718a9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d47a5a77-8d4f-401b-b4a3-80c2d27718a9-openshift-service-ca\") pod \"d47a5a77-8d4f-401b-b4a3-80c2d27718a9\" (UID: \"d47a5a77-8d4f-401b-b4a3-80c2d27718a9\") "
Oct 30 17:25:43.525684 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:43.525615    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d47a5a77-8d4f-401b-b4a3-80c2d27718a9-openshift-service-ca" pod "d47a5a77-8d4f-401b-b4a3-80c2d27718a9" (UID: "d47a5a77-8d4f-401b-b4a3-80c2d27718a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:25:43.527239 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:43.527211    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d47a5a77-8d4f-401b-b4a3-80c2d27718a9-metrics-certs" pod "d47a5a77-8d4f-401b-b4a3-80c2d27718a9" (UID: "d47a5a77-8d4f-401b-b4a3-80c2d27718a9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:25:43.527353 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:43.527246    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d47a5a77-8d4f-401b-b4a3-80c2d27718a9-maas-api-tls" pod "d47a5a77-8d4f-401b-b4a3-80c2d27718a9" (UID: "d47a5a77-8d4f-401b-b4a3-80c2d27718a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:25:43.527868 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:43.527841    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d47a5a77-8d4f-401b-b4a3-80c2d27718a9-kube-api-access-zt744" pod "d47a5a77-8d4f-401b-b4a3-80c2d27718a9" (UID: "d47a5a77-8d4f-401b-b4a3-80c2d27718a9"). InnerVolumeSpecName "kube-api-access-zt744". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:25:43.626747 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:43.626658    4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d47a5a77-8d4f-401b-b4a3-80c2d27718a9-openshift-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:25:43.626747 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:43.626690    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt744\" (UniqueName: \"kubernetes.io/projected/d47a5a77-8d4f-401b-b4a3-80c2d27718a9-kube-api-access-zt744\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:25:43.626747 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:43.626700    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d47a5a77-8d4f-401b-b4a3-80c2d27718a9-maas-api-tls\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:25:43.626747 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:43.626710    4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d47a5a77-8d4f-401b-b4a3-80c2d27718a9-metrics-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:25:44.165901 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:44.165871    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr"
Oct 30 17:25:44.166328 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:44.165901    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr" event={"ID":"d47a5a77-8d4f-401b-b4a3-80c2d27718a9","Type":"ContainerDied","Data":"587de0d7512fb83e44663aa22f20046450d1b34a66fdfb77e3bd242d301fe1d7"}
Oct 30 17:25:44.166328 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:44.165962    4433 scope.go:122] "RemoveContainer" containerID="54774d1a67235481922c6a7195c597c91ec65844830283a4bf49c8eb5a6ca5b9"
Oct 30 17:25:44.216645 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:44.216609    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr"]
Oct 30 17:25:44.227638 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:44.227610    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-42d381-7f5d5c5549-dhmhr"]
Oct 30 17:25:45.696047 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:45.695943    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d47a5a77-8d4f-401b-b4a3-80c2d27718a9" path="/var/lib/kubelet/pods/d47a5a77-8d4f-401b-b4a3-80c2d27718a9/volumes"
Oct 30 17:25:47.580145 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:47.580102    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd"]
Oct 30 17:25:47.581834 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:47.580966    4433 cpu_manager.go:414] "containerMap: removing container" podUID="d47a5a77-8d4f-401b-b4a3-80c2d27718a9" containerName="maas-api"
Oct 30 17:25:47.581834 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:47.580988    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d47a5a77-8d4f-401b-b4a3-80c2d27718a9" containerName="maas-api"
Oct 30 17:25:47.581834 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:47.581005    4433 cpu_manager.go:414] "containerMap: removing container" podUID="1ca3e5af-f7e2-47d5-bc76-66d298b16971" containerName="main"
Oct 30 17:25:47.581834 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:47.581011    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ca3e5af-f7e2-47d5-bc76-66d298b16971" containerName="main"
Oct 30 17:25:47.581834 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:47.581191    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="d47a5a77-8d4f-401b-b4a3-80c2d27718a9" containerName="maas-api"
Oct 30 17:25:47.581834 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:47.581208    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ca3e5af-f7e2-47d5-bc76-66d298b16971" containerName="main"
Oct 30 17:25:47.596897 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:47.596873    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd"
Oct 30 17:25:47.614858 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:47.610849    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd"]
Oct 30 17:25:47.614858 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:47.611665    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-a43843\"/\"openshift-service-ca.crt\""
Oct 30 17:25:47.622742 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:47.622717    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-a43843\"/\"default-dockercfg-xg5wc\""
Oct 30 17:25:47.622876 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:47.622741    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-a43843\"/\"auth-test-model-a43843-kserve-self-signed-certs\""
Oct 30 17:25:47.681964 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:47.681923    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1921735-27bd-46c9-9885-879a85d541a9-model-cache\") pod \"auth-test-model-a43843-kserve-68f64f886f-69xpd\" (UID: \"b1921735-27bd-46c9-9885-879a85d541a9\") " pod="ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd"
Oct 30 17:25:47.682181 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:47.682048    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1921735-27bd-46c9-9885-879a85d541a9-tmp-dir\") pod \"auth-test-model-a43843-kserve-68f64f886f-69xpd\" (UID: \"b1921735-27bd-46c9-9885-879a85d541a9\") " pod="ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd"
Oct 30 17:25:47.682181 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:47.682102    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1921735-27bd-46c9-9885-879a85d541a9-dshm\") pod \"auth-test-model-a43843-kserve-68f64f886f-69xpd\" (UID: \"b1921735-27bd-46c9-9885-879a85d541a9\") " pod="ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd"
Oct 30 17:25:47.682300 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:47.682196    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1921735-27bd-46c9-9885-879a85d541a9-home\") pod \"auth-test-model-a43843-kserve-68f64f886f-69xpd\" (UID: \"b1921735-27bd-46c9-9885-879a85d541a9\") " pod="ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd"
Oct 30 17:25:47.682300 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:47.682254    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1921735-27bd-46c9-9885-879a85d541a9-tls-certs\") pod \"auth-test-model-a43843-kserve-68f64f886f-69xpd\" (UID: \"b1921735-27bd-46c9-9885-879a85d541a9\") " pod="ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd"
Oct 30 17:25:47.682427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:47.682383    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9lvs\" (UniqueName: \"kubernetes.io/projected/b1921735-27bd-46c9-9885-879a85d541a9-kube-api-access-d9lvs\") pod \"auth-test-model-a43843-kserve-68f64f886f-69xpd\" (UID: \"b1921735-27bd-46c9-9885-879a85d541a9\") " pod="ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd"
Oct 30 17:25:47.712125 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:47.712089    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-a43843\"/\"kube-root-ca.crt\""
Oct 30 17:25:47.783906 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:47.783855    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1921735-27bd-46c9-9885-879a85d541a9-model-cache\") pod \"auth-test-model-a43843-kserve-68f64f886f-69xpd\" (UID: \"b1921735-27bd-46c9-9885-879a85d541a9\") " pod="ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd"
Oct 30 17:25:47.784110 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:47.783938    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1921735-27bd-46c9-9885-879a85d541a9-tmp-dir\") pod \"auth-test-model-a43843-kserve-68f64f886f-69xpd\" (UID: \"b1921735-27bd-46c9-9885-879a85d541a9\") " pod="ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd"
Oct 30 17:25:47.784110 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:47.784002    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1921735-27bd-46c9-9885-879a85d541a9-dshm\") pod \"auth-test-model-a43843-kserve-68f64f886f-69xpd\" (UID: \"b1921735-27bd-46c9-9885-879a85d541a9\") " pod="ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd"
Oct 30 17:25:47.784110 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:47.784077    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1921735-27bd-46c9-9885-879a85d541a9-home\") pod \"auth-test-model-a43843-kserve-68f64f886f-69xpd\" (UID: \"b1921735-27bd-46c9-9885-879a85d541a9\") " pod="ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd"
Oct 30 17:25:47.784309 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:47.784210    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1921735-27bd-46c9-9885-879a85d541a9-tls-certs\") pod \"auth-test-model-a43843-kserve-68f64f886f-69xpd\" (UID: \"b1921735-27bd-46c9-9885-879a85d541a9\") " pod="ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd"
Oct 30 17:25:47.784378 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:47.784310    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9lvs\" (UniqueName: \"kubernetes.io/projected/b1921735-27bd-46c9-9885-879a85d541a9-kube-api-access-d9lvs\") pod \"auth-test-model-a43843-kserve-68f64f886f-69xpd\" (UID: \"b1921735-27bd-46c9-9885-879a85d541a9\") " pod="ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd"
Oct 30 17:25:47.784483 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:47.784383    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1921735-27bd-46c9-9885-879a85d541a9-model-cache\") pod \"auth-test-model-a43843-kserve-68f64f886f-69xpd\" (UID: \"b1921735-27bd-46c9-9885-879a85d541a9\") " pod="ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd"
Oct 30 17:25:47.784483 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:47.784445    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1921735-27bd-46c9-9885-879a85d541a9-tmp-dir\") pod \"auth-test-model-a43843-kserve-68f64f886f-69xpd\" (UID: \"b1921735-27bd-46c9-9885-879a85d541a9\") " pod="ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd"
Oct 30 17:25:47.785199 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:47.785173    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1921735-27bd-46c9-9885-879a85d541a9-home\") pod \"auth-test-model-a43843-kserve-68f64f886f-69xpd\" (UID: \"b1921735-27bd-46c9-9885-879a85d541a9\") " pod="ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd"
Oct 30 17:25:47.787033 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:47.787006    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1921735-27bd-46c9-9885-879a85d541a9-dshm\") pod \"auth-test-model-a43843-kserve-68f64f886f-69xpd\" (UID: \"b1921735-27bd-46c9-9885-879a85d541a9\") " pod="ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd"
Oct 30 17:25:47.787387 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:47.787362    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1921735-27bd-46c9-9885-879a85d541a9-tls-certs\") pod \"auth-test-model-a43843-kserve-68f64f886f-69xpd\" (UID: \"b1921735-27bd-46c9-9885-879a85d541a9\") " pod="ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd"
Oct 30 17:25:47.816135 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:47.816067    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9lvs\" (UniqueName: \"kubernetes.io/projected/b1921735-27bd-46c9-9885-879a85d541a9-kube-api-access-d9lvs\") pod \"auth-test-model-a43843-kserve-68f64f886f-69xpd\" (UID: \"b1921735-27bd-46c9-9885-879a85d541a9\") " pod="ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd"
Oct 30 17:25:47.914525 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:47.913617    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd"
Oct 30 17:25:48.253136 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:48.253097    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd"]
Oct 30 17:25:48.255181 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:25:48.255151    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1921735_27bd_46c9_9885_879a85d541a9.slice/crio-83ab6312c03efb8224bd8fb781700968eed542804c5b0e34c4206df266b88afe WatchSource:0}: Error finding container 83ab6312c03efb8224bd8fb781700968eed542804c5b0e34c4206df266b88afe: Status 404 returned error can't find the container with id 83ab6312c03efb8224bd8fb781700968eed542804c5b0e34c4206df266b88afe
Oct 30 17:25:49.200056 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:49.200022    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd" event={"ID":"b1921735-27bd-46c9-9885-879a85d541a9","Type":"ContainerStarted","Data":"0eb9160443c009aa252755df9031dae8a5ddf5b92fa1b6176deba22ab3e11999"}
Oct 30 17:25:49.200566 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:49.200064    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd" event={"ID":"b1921735-27bd-46c9-9885-879a85d541a9","Type":"ContainerStarted","Data":"83ab6312c03efb8224bd8fb781700968eed542804c5b0e34c4206df266b88afe"}
Oct 30 17:25:49.242054 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:49.241985    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd" podStartSLOduration=2.241967071 podStartE2EDuration="2.241967071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:25:49.240479188 +0000 UTC m=+2325.675381398" watchObservedRunningTime="2026-10-30 17:25:49.241967071 +0000 UTC m=+2325.676869278"
Oct 30 17:25:49.914507 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:49.914472    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd"
Oct 30 17:25:52.011779 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:52.011743    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:25:55.641520 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:55.641475    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk"]
Oct 30 17:25:55.645828 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:55.645804    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk"
Oct 30 17:25:55.652919 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:55.652889    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9z8s\" (UniqueName: \"kubernetes.io/projected/8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545-kube-api-access-l9z8s\") pod \"maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk\" (UID: \"8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk"
Oct 30 17:25:55.653080 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:55.652932    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk\" (UID: \"8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk"
Oct 30 17:25:55.653080 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:55.653050    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545-maas-api-tls\") pod \"maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk\" (UID: \"8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk"
Oct 30 17:25:55.653196 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:55.653096    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545-metrics-certs\") pod \"maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk\" (UID: \"8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk"
Oct 30 17:25:55.671756 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:55.671720    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-abe243-serving-cert\""
Oct 30 17:25:55.675482 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:55.675451    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk"]
Oct 30 17:25:55.754647 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:55.754615    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9z8s\" (UniqueName: \"kubernetes.io/projected/8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545-kube-api-access-l9z8s\") pod \"maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk\" (UID: \"8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk"
Oct 30 17:25:55.754821 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:55.754655    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk\" (UID: \"8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk"
Oct 30 17:25:55.754821 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:55.754712    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545-maas-api-tls\") pod \"maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk\" (UID: \"8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk"
Oct 30 17:25:55.754821 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:55.754742    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545-metrics-certs\") pod \"maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk\" (UID: \"8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk"
Oct 30 17:25:55.754956 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:25:55.754838    4433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-abe243-serving-cert: secret "maas-api-e2e-ipp-a-abe243-serving-cert" not found
Oct 30 17:25:55.754956 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:25:55.754910    4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545-maas-api-tls podName:8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545 nodeName:}" failed. No retries permitted until 2026-10-30 17:25:56.254891583 +0000 UTC m=+2332.689793782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545-maas-api-tls") pod "maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk" (UID: "8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545") : secret "maas-api-e2e-ipp-a-abe243-serving-cert" not found
Oct 30 17:25:55.755372 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:55.755351    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk\" (UID: \"8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk"
Oct 30 17:25:55.757144 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:55.757120    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545-metrics-certs\") pod \"maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk\" (UID: \"8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk"
Oct 30 17:25:55.779509 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:55.779479    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9z8s\" (UniqueName: \"kubernetes.io/projected/8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545-kube-api-access-l9z8s\") pod \"maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk\" (UID: \"8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk"
Oct 30 17:25:56.261885 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:56.261847    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545-maas-api-tls\") pod \"maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk\" (UID: \"8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk"
Oct 30 17:25:56.264381 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:56.264353    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545-maas-api-tls\") pod \"maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk\" (UID: \"8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk"
Oct 30 17:25:56.560088 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:56.559992    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk"
Oct 30 17:25:56.895217 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:56.895140    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk"]
Oct 30 17:25:56.899231 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:25:56.899194    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd302f4_10c7_4e0b_a5f5_fd3ea26d2545.slice/crio-665befa699347cfeaced234853ed9dc9f00de651cfeaf166c1565dc02ed44d16 WatchSource:0}: Error finding container 665befa699347cfeaced234853ed9dc9f00de651cfeaf166c1565dc02ed44d16: Status 404 returned error can't find the container with id 665befa699347cfeaced234853ed9dc9f00de651cfeaf166c1565dc02ed44d16
Oct 30 17:25:56.900943 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:56.900919    4433 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 17:25:56.979433 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:25:56.979383    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:25:57.285131 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:57.284955    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk" event={"ID":"8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545","Type":"ContainerStarted","Data":"665befa699347cfeaced234853ed9dc9f00de651cfeaf166c1565dc02ed44d16"}
Oct 30 17:25:57.914521 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:57.914488    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd"
Oct 30 17:25:57.927193 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:57.927165    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd"
Oct 30 17:25:58.292892 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:58.292857    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk" event={"ID":"8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545","Type":"ContainerStarted","Data":"f151bad01a5375492112caaedb43662cfdffe673c14d62560eee30c9b4083341"}
Oct 30 17:25:58.293090 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:58.293069    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk"
Oct 30 17:25:58.304723 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:58.304691    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd"
Oct 30 17:25:58.337705 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:25:58.337647    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk" podStartSLOduration=2.944466298 podStartE2EDuration="3.33763051s" totalImagesPullingTime="393.164212ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:25:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:25:58.334529896 +0000 UTC m=+2334.769432102" watchObservedRunningTime="2026-10-30 17:25:58.33763051 +0000 UTC m=+2334.772532717"
Oct 30 17:26:04.306253 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:04.306222    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk"
Oct 30 17:26:26.979573 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:26:26.979533    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:26:28.860623 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:28.860579    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd"]
Oct 30 17:26:36.426288 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:36.426242    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-76f8ff444c-gtjjg"]
Oct 30 17:26:41.430316 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:41.430243    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-76f8ff444c-gtjjg" podUID="3d2fff10-15e7-48da-abfd-b3897f0ccba4" containerName="payload-pre-processing" containerID="cri-o://88e5fb7abcfbcfdc599be86cf172d396c1d064191c8bb9dc2ef132945eb4e600" gracePeriod=25
Oct 30 17:26:41.624737 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:41.624697    4433 generic.go:356] "Generic (PLEG): container finished" podID="3d2fff10-15e7-48da-abfd-b3897f0ccba4" containerID="88e5fb7abcfbcfdc599be86cf172d396c1d064191c8bb9dc2ef132945eb4e600" exitCode=0
Oct 30 17:26:41.625126 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:41.625097    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-76f8ff444c-gtjjg" event={"ID":"3d2fff10-15e7-48da-abfd-b3897f0ccba4","Type":"ContainerDied","Data":"88e5fb7abcfbcfdc599be86cf172d396c1d064191c8bb9dc2ef132945eb4e600"}
Oct 30 17:26:41.693573 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:41.693476    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:26:42.121934 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:42.121898    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-76f8ff444c-gtjjg"
Oct 30 17:26:42.270782 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:42.270273    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d2fff10-15e7-48da-abfd-b3897f0ccba4-kube-api-access-svq9j\" (UniqueName: \"kubernetes.io/projected/3d2fff10-15e7-48da-abfd-b3897f0ccba4-kube-api-access-svq9j\") pod \"3d2fff10-15e7-48da-abfd-b3897f0ccba4\" (UID: \"3d2fff10-15e7-48da-abfd-b3897f0ccba4\") "
Oct 30 17:26:42.270984 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:42.270926    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d2fff10-15e7-48da-abfd-b3897f0ccba4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d2fff10-15e7-48da-abfd-b3897f0ccba4-plugins-config-volume\") pod \"3d2fff10-15e7-48da-abfd-b3897f0ccba4\" (UID: \"3d2fff10-15e7-48da-abfd-b3897f0ccba4\") "
Oct 30 17:26:42.296488 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:42.289674    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d2fff10-15e7-48da-abfd-b3897f0ccba4-plugins-config-volume" pod "3d2fff10-15e7-48da-abfd-b3897f0ccba4" (UID: "3d2fff10-15e7-48da-abfd-b3897f0ccba4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:26:42.319761 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:42.319715    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2fff10-15e7-48da-abfd-b3897f0ccba4-kube-api-access-svq9j" pod "3d2fff10-15e7-48da-abfd-b3897f0ccba4" (UID: "3d2fff10-15e7-48da-abfd-b3897f0ccba4"). InnerVolumeSpecName "kube-api-access-svq9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:26:42.375495 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:42.375453    4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d2fff10-15e7-48da-abfd-b3897f0ccba4-plugins-config-volume\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:26:42.375742 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:42.375722    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svq9j\" (UniqueName: \"kubernetes.io/projected/3d2fff10-15e7-48da-abfd-b3897f0ccba4-kube-api-access-svq9j\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:26:42.632960 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:42.632880    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-76f8ff444c-gtjjg"
Oct 30 17:26:42.632960 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:42.632914    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-76f8ff444c-gtjjg" event={"ID":"3d2fff10-15e7-48da-abfd-b3897f0ccba4","Type":"ContainerDied","Data":"38afd138bcd8cfd1c60e23683af188ca35ebb2d5b9ef1f5fe0aff959c89a44b4"}
Oct 30 17:26:42.633465 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:42.632977    4433 scope.go:122] "RemoveContainer" containerID="88e5fb7abcfbcfdc599be86cf172d396c1d064191c8bb9dc2ef132945eb4e600"
Oct 30 17:26:42.690669 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:42.690539    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-76f8ff444c-gtjjg"]
Oct 30 17:26:42.701669 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:42.701636    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-76f8ff444c-gtjjg"]
Oct 30 17:26:43.697384 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:43.697351    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d2fff10-15e7-48da-abfd-b3897f0ccba4" path="/var/lib/kubelet/pods/3d2fff10-15e7-48da-abfd-b3897f0ccba4/volumes"
Oct 30 17:26:43.897692 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:43.897646    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd" podUID="b1921735-27bd-46c9-9885-879a85d541a9" containerName="main" containerID="cri-o://0eb9160443c009aa252755df9031dae8a5ddf5b92fa1b6176deba22ab3e11999" gracePeriod=45
Oct 30 17:26:44.299131 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:44.299069    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd"
Oct 30 17:26:44.331625 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:44.331590    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1921735-27bd-46c9-9885-879a85d541a9-kube-api-access-d9lvs\" (UniqueName: \"kubernetes.io/projected/b1921735-27bd-46c9-9885-879a85d541a9-kube-api-access-d9lvs\") pod \"b1921735-27bd-46c9-9885-879a85d541a9\" (UID: \"b1921735-27bd-46c9-9885-879a85d541a9\") "
Oct 30 17:26:44.331823 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:44.331654    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1921735-27bd-46c9-9885-879a85d541a9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1921735-27bd-46c9-9885-879a85d541a9-tmp-dir\") pod \"b1921735-27bd-46c9-9885-879a85d541a9\" (UID: \"b1921735-27bd-46c9-9885-879a85d541a9\") "
Oct 30 17:26:44.331823 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:44.331689    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1921735-27bd-46c9-9885-879a85d541a9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1921735-27bd-46c9-9885-879a85d541a9-dshm\") pod \"b1921735-27bd-46c9-9885-879a85d541a9\" (UID: \"b1921735-27bd-46c9-9885-879a85d541a9\") "
Oct 30 17:26:44.331969 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:44.331914    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1921735-27bd-46c9-9885-879a85d541a9-home\" (UniqueName: \"kubernetes.io/empty-dir/b1921735-27bd-46c9-9885-879a85d541a9-home\") pod \"b1921735-27bd-46c9-9885-879a85d541a9\" (UID: \"b1921735-27bd-46c9-9885-879a85d541a9\") "
Oct 30 17:26:44.332026 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:44.331963    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1921735-27bd-46c9-9885-879a85d541a9-tmp-dir" pod "b1921735-27bd-46c9-9885-879a85d541a9" (UID: "b1921735-27bd-46c9-9885-879a85d541a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:26:44.332026 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:44.332016    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1921735-27bd-46c9-9885-879a85d541a9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1921735-27bd-46c9-9885-879a85d541a9-model-cache\") pod \"b1921735-27bd-46c9-9885-879a85d541a9\" (UID: \"b1921735-27bd-46c9-9885-879a85d541a9\") "
Oct 30 17:26:44.332129 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:44.332114    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1921735-27bd-46c9-9885-879a85d541a9-tls-certs\" (UniqueName: \"kubernetes.io/secret/b1921735-27bd-46c9-9885-879a85d541a9-tls-certs\") pod \"b1921735-27bd-46c9-9885-879a85d541a9\" (UID: \"b1921735-27bd-46c9-9885-879a85d541a9\") "
Oct 30 17:26:44.332237 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:44.332189    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1921735-27bd-46c9-9885-879a85d541a9-home" pod "b1921735-27bd-46c9-9885-879a85d541a9" (UID: "b1921735-27bd-46c9-9885-879a85d541a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:26:44.332394 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:44.332282    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1921735-27bd-46c9-9885-879a85d541a9-model-cache" pod "b1921735-27bd-46c9-9885-879a85d541a9" (UID: "b1921735-27bd-46c9-9885-879a85d541a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:26:44.332734 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:44.332704    4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1921735-27bd-46c9-9885-879a85d541a9-home\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:26:44.332734 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:44.332731    4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1921735-27bd-46c9-9885-879a85d541a9-model-cache\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:26:44.332878 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:44.332746    4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1921735-27bd-46c9-9885-879a85d541a9-tmp-dir\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:26:44.334252 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:44.334227    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1921735-27bd-46c9-9885-879a85d541a9-dshm" pod "b1921735-27bd-46c9-9885-879a85d541a9" (UID: "b1921735-27bd-46c9-9885-879a85d541a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:26:44.334495 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:44.334473    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1921735-27bd-46c9-9885-879a85d541a9-tls-certs" pod "b1921735-27bd-46c9-9885-879a85d541a9" (UID: "b1921735-27bd-46c9-9885-879a85d541a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:26:44.334495 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:44.334486    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1921735-27bd-46c9-9885-879a85d541a9-kube-api-access-d9lvs" pod "b1921735-27bd-46c9-9885-879a85d541a9" (UID: "b1921735-27bd-46c9-9885-879a85d541a9"). InnerVolumeSpecName "kube-api-access-d9lvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:26:44.434305 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:44.434254    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9lvs\" (UniqueName: \"kubernetes.io/projected/b1921735-27bd-46c9-9885-879a85d541a9-kube-api-access-d9lvs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:26:44.434305 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:44.434297    4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1921735-27bd-46c9-9885-879a85d541a9-dshm\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:26:44.434305 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:44.434312    4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1921735-27bd-46c9-9885-879a85d541a9-tls-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:26:44.677436 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:44.676696    4433 generic.go:356] "Generic (PLEG): container finished" podID="b1921735-27bd-46c9-9885-879a85d541a9" containerID="0eb9160443c009aa252755df9031dae8a5ddf5b92fa1b6176deba22ab3e11999" exitCode=0
Oct 30 17:26:44.677436 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:44.676811    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd" event={"ID":"b1921735-27bd-46c9-9885-879a85d541a9","Type":"ContainerDied","Data":"0eb9160443c009aa252755df9031dae8a5ddf5b92fa1b6176deba22ab3e11999"}
Oct 30 17:26:44.677436 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:44.676856    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd" event={"ID":"b1921735-27bd-46c9-9885-879a85d541a9","Type":"ContainerDied","Data":"83ab6312c03efb8224bd8fb781700968eed542804c5b0e34c4206df266b88afe"}
Oct 30 17:26:44.677436 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:44.676880    4433 scope.go:122] "RemoveContainer" containerID="0eb9160443c009aa252755df9031dae8a5ddf5b92fa1b6176deba22ab3e11999"
Oct 30 17:26:44.677436 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:44.677052    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd"
Oct 30 17:26:44.698042 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:44.698014    4433 scope.go:122] "RemoveContainer" containerID="0eb9160443c009aa252755df9031dae8a5ddf5b92fa1b6176deba22ab3e11999"
Oct 30 17:26:44.698466 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:26:44.698444    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eb9160443c009aa252755df9031dae8a5ddf5b92fa1b6176deba22ab3e11999\": container with ID starting with 0eb9160443c009aa252755df9031dae8a5ddf5b92fa1b6176deba22ab3e11999 not found: ID does not exist" containerID="0eb9160443c009aa252755df9031dae8a5ddf5b92fa1b6176deba22ab3e11999"
Oct 30 17:26:44.698517 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:44.698480    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eb9160443c009aa252755df9031dae8a5ddf5b92fa1b6176deba22ab3e11999"} err="failed to get container status \"0eb9160443c009aa252755df9031dae8a5ddf5b92fa1b6176deba22ab3e11999\": rpc error: code = NotFound desc = could not find container \"0eb9160443c009aa252755df9031dae8a5ddf5b92fa1b6176deba22ab3e11999\": container with ID starting with 0eb9160443c009aa252755df9031dae8a5ddf5b92fa1b6176deba22ab3e11999 not found: ID does not exist"
Oct 30 17:26:44.736199 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:44.736161    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd"]
Oct 30 17:26:44.749201 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:44.749166    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd"]
Oct 30 17:26:45.294690 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:45.294641    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-a43843/auth-test-model-a43843-kserve-68f64f886f-69xpd" podUID="b1921735-27bd-46c9-9885-879a85d541a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.143:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:26:45.695655 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:45.695618    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1921735-27bd-46c9-9885-879a85d541a9" path="/var/lib/kubelet/pods/b1921735-27bd-46c9-9885-879a85d541a9/volumes"
Oct 30 17:26:50.718529 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:50.718491    4433 generic.go:356] "Generic (PLEG): container finished" podID="d74c8b0b-844e-493f-a994-5b83b7b73589" containerID="890ea6dad27d4f3b9958d2f10ba005dad229051350aa4f19a895e1390d0d8954" exitCode=0
Oct 30 17:26:50.719040 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:50.718567    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww" event={"ID":"d74c8b0b-844e-493f-a994-5b83b7b73589","Type":"ContainerDied","Data":"890ea6dad27d4f3b9958d2f10ba005dad229051350aa4f19a895e1390d0d8954"}
Oct 30 17:26:50.719040 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:50.718995    4433 scope.go:122] "RemoveContainer" containerID="890ea6dad27d4f3b9958d2f10ba005dad229051350aa4f19a895e1390d0d8954"
Oct 30 17:26:50.720837 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:50.720810    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:26:50.720940 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:50.720882    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:26:50.720940 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:50.720914    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:26:51.726066 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:51.726030    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww" event={"ID":"d74c8b0b-844e-493f-a994-5b83b7b73589","Type":"ContainerStarted","Data":"44e0bc3489b057ec0802fdc99a284108a00a6fbbfa314825aa02d2e6300a09c1"}
Oct 30 17:26:52.422595 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:52.422560    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:26:52.422776 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:52.422753    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 17:26:52.422825 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:52.422811    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww" podUID="d74c8b0b-844e-493f-a994-5b83b7b73589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 17:26:53.422556 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:53.422514    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 17:26:53.422938 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:53.422573    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww" podUID="d74c8b0b-844e-493f-a994-5b83b7b73589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 17:26:54.423326 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:54.423291    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 17:26:54.423753 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:54.423345    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww" podUID="d74c8b0b-844e-493f-a994-5b83b7b73589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 17:26:55.422890 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.422851    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 17:26:55.423065 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.422914    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww" podUID="d74c8b0b-844e-493f-a994-5b83b7b73589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 17:26:55.492820 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.491745    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"]
Oct 30 17:26:55.492820 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.492645    4433 cpu_manager.go:414] "containerMap: removing container" podUID="3d2fff10-15e7-48da-abfd-b3897f0ccba4" containerName="payload-pre-processing"
Oct 30 17:26:55.492820 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.492669    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d2fff10-15e7-48da-abfd-b3897f0ccba4" containerName="payload-pre-processing"
Oct 30 17:26:55.492820 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.492703    4433 cpu_manager.go:414] "containerMap: removing container" podUID="b1921735-27bd-46c9-9885-879a85d541a9" containerName="main"
Oct 30 17:26:55.492820 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.492712    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1921735-27bd-46c9-9885-879a85d541a9" containerName="main"
Oct 30 17:26:55.493630 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.492894    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d2fff10-15e7-48da-abfd-b3897f0ccba4" containerName="payload-pre-processing"
Oct 30 17:26:55.493630 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.492913    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1921735-27bd-46c9-9885-879a85d541a9" containerName="main"
Oct 30 17:26:55.502335 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.501682    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:26:55.518398 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.518370    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"]
Oct 30 17:26:55.546658 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.546627    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-75586d-openshift-default-dockercfg-kkkg2\""
Oct 30 17:26:55.632013 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.631955    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-workload-certs\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:26:55.632315 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.632291    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-credential-socket\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:26:55.632477 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.632457    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-istio-token\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:26:55.632698 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.632682    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-istiod-ca-cert\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:26:55.632876 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.632862    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-istio-envoy\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:26:55.632986 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.632972    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-istio-podinfo\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:26:55.633093 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.633078    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42c9j\" (UniqueName: \"kubernetes.io/projected/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-kube-api-access-42c9j\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:26:55.633209 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.633195    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-workload-socket\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:26:55.633358 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.633345    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-istio-data\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:26:55.736732 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.736657    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-istiod-ca-cert\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:26:55.764656 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.764614    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-istiod-ca-cert\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:26:55.765795 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.765770    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-istio-envoy\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:26:55.766398 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.766377    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-istio-podinfo\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:26:55.767503 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.767485    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42c9j\" (UniqueName: \"kubernetes.io/projected/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-kube-api-access-42c9j\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:26:55.768035 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.768014    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-workload-socket\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:26:55.768628 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.768464    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-workload-socket\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:26:55.769977 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.769929    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-istio-data\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:26:55.783891 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.782530    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-istio-data\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:26:55.783891 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.782656    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-workload-certs\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:26:55.783891 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.782745    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-credential-socket\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:26:55.783891 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.782783    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-istio-token\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:26:55.783891 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.783567    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-workload-certs\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:26:55.783891 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.783832    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-credential-socket\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:26:55.803746 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.803681    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42c9j\" (UniqueName: \"kubernetes.io/projected/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-kube-api-access-42c9j\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:26:55.816307 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.816234    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-istio-envoy\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:26:55.829978 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.817381    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-istio-podinfo\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:26:55.831049 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:55.830690    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-istio-token\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:26:56.121218 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:56.121127    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:26:56.423166 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:56.423086    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 17:26:56.423166 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:56.423146    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww" podUID="d74c8b0b-844e-493f-a994-5b83b7b73589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 17:26:56.496480 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:56.496455    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"]
Oct 30 17:26:56.501578 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:26:56.501545    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaefefaea_deba_44ec_a357_4fb4ba5e7f6e.slice/crio-61868b56164b880d11949e735a23aee3996c2e6dad98bffab2ea29a245be0bfb WatchSource:0}: Error finding container 61868b56164b880d11949e735a23aee3996c2e6dad98bffab2ea29a245be0bfb: Status 404 returned error can't find the container with id 61868b56164b880d11949e735a23aee3996c2e6dad98bffab2ea29a245be0bfb
Oct 30 17:26:56.504890 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:56.504852    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:26:56.504967 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:56.504939    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:26:56.505007 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:56.504982    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:26:56.809648 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:56.809610    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb" event={"ID":"aefefaea-deba-44ec-a357-4fb4ba5e7f6e","Type":"ContainerStarted","Data":"284070c3fc1cda348dc8dd123a9fd8793da2aea27df2249862097a86b81ea93d"}
Oct 30 17:26:56.809827 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:56.809679    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb" event={"ID":"aefefaea-deba-44ec-a357-4fb4ba5e7f6e","Type":"ContainerStarted","Data":"61868b56164b880d11949e735a23aee3996c2e6dad98bffab2ea29a245be0bfb"}
Oct 30 17:26:56.860438 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:56.859533    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb" podStartSLOduration=1.859515544 podStartE2EDuration="1.859515544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:56.858702632 +0000 UTC m=+2393.293604839" watchObservedRunningTime="2026-10-30 17:26:56.859515544 +0000 UTC m=+2393.294417748"
Oct 30 17:26:57.036602 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:26:57.036562    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:26:57.122199 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:57.122115    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:26:57.123435 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:57.123382    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 17:26:57.123560 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:57.123449    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb" podUID="aefefaea-deba-44ec-a357-4fb4ba5e7f6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 17:26:57.423002 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:57.422920    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 17:26:57.423002 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:57.422972    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww" podUID="d74c8b0b-844e-493f-a994-5b83b7b73589" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 17:26:58.135380 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:58.135343    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:26:58.427569 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:58.427534    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:26:58.427847 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:58.427822    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:26:58.428701 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:58.428680    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:26:58.824768 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:58.824683    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:26:58.826136 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:26:58.826116    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:27:00.088071 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.088037    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"]
Oct 30 17:27:00.092605 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.092582    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"
Oct 30 17:27:00.109223 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.109197    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-db8417-openshift-default-dockercfg-94mls\""
Oct 30 17:27:00.124662 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.124626    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"]
Oct 30 17:27:00.166589 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.166549    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61e37504-76d8-47bf-8fa8-1a66d794e465-istio-podinfo\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"
Oct 30 17:27:00.166781 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.166594    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61e37504-76d8-47bf-8fa8-1a66d794e465-istio-data\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"
Oct 30 17:27:00.166781 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.166677    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61e37504-76d8-47bf-8fa8-1a66d794e465-credential-socket\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"
Oct 30 17:27:00.166911 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.166828    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61e37504-76d8-47bf-8fa8-1a66d794e465-istiod-ca-cert\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"
Oct 30 17:27:00.166911 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.166859    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltftd\" (UniqueName: \"kubernetes.io/projected/61e37504-76d8-47bf-8fa8-1a66d794e465-kube-api-access-ltftd\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"
Oct 30 17:27:00.166990 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.166936    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61e37504-76d8-47bf-8fa8-1a66d794e465-workload-socket\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"
Oct 30 17:27:00.167035 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.167025    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61e37504-76d8-47bf-8fa8-1a66d794e465-workload-certs\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"
Oct 30 17:27:00.167075 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.167055    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61e37504-76d8-47bf-8fa8-1a66d794e465-istio-token\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"
Oct 30 17:27:00.167120 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.167086    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61e37504-76d8-47bf-8fa8-1a66d794e465-istio-envoy\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"
Oct 30 17:27:00.268005 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.267963    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61e37504-76d8-47bf-8fa8-1a66d794e465-istiod-ca-cert\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"
Oct 30 17:27:00.268005 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.268003    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltftd\" (UniqueName: \"kubernetes.io/projected/61e37504-76d8-47bf-8fa8-1a66d794e465-kube-api-access-ltftd\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"
Oct 30 17:27:00.268255 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.268032    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61e37504-76d8-47bf-8fa8-1a66d794e465-workload-socket\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"
Oct 30 17:27:00.268255 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.268101    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61e37504-76d8-47bf-8fa8-1a66d794e465-workload-certs\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"
Oct 30 17:27:00.268255 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.268136    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61e37504-76d8-47bf-8fa8-1a66d794e465-istio-token\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"
Oct 30 17:27:00.268255 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.268184    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61e37504-76d8-47bf-8fa8-1a66d794e465-istio-envoy\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"
Oct 30 17:27:00.268523 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.268287    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61e37504-76d8-47bf-8fa8-1a66d794e465-istio-podinfo\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"
Oct 30 17:27:00.268523 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.268315    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61e37504-76d8-47bf-8fa8-1a66d794e465-istio-data\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"
Oct 30 17:27:00.268523 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.268345    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61e37504-76d8-47bf-8fa8-1a66d794e465-credential-socket\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"
Oct 30 17:27:00.268692 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.268629    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61e37504-76d8-47bf-8fa8-1a66d794e465-workload-certs\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"
Oct 30 17:27:00.268756 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.268738    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61e37504-76d8-47bf-8fa8-1a66d794e465-workload-socket\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"
Oct 30 17:27:00.268838 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.268796    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61e37504-76d8-47bf-8fa8-1a66d794e465-istio-data\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"
Oct 30 17:27:00.268910 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.268816    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61e37504-76d8-47bf-8fa8-1a66d794e465-credential-socket\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"
Oct 30 17:27:00.268910 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.268862    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61e37504-76d8-47bf-8fa8-1a66d794e465-istiod-ca-cert\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"
Oct 30 17:27:00.271097 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.271044    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61e37504-76d8-47bf-8fa8-1a66d794e465-istio-envoy\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"
Oct 30 17:27:00.271689 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.271667    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61e37504-76d8-47bf-8fa8-1a66d794e465-istio-podinfo\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"
Oct 30 17:27:00.292135 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.292106    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61e37504-76d8-47bf-8fa8-1a66d794e465-istio-token\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"
Oct 30 17:27:00.293774 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.293749    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltftd\" (UniqueName: \"kubernetes.io/projected/61e37504-76d8-47bf-8fa8-1a66d794e465-kube-api-access-ltftd\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"
Oct 30 17:27:00.407739 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.407641    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"
Oct 30 17:27:00.753091 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.753061    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"]
Oct 30 17:27:00.755042 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:27:00.755016    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e37504_76d8_47bf_8fa8_1a66d794e465.slice/crio-4965466a76266e3e6d8ab3e3c8870a53ed9526ec0a2c81c0ddbfae8728e50d32 WatchSource:0}: Error finding container 4965466a76266e3e6d8ab3e3c8870a53ed9526ec0a2c81c0ddbfae8728e50d32: Status 404 returned error can't find the container with id 4965466a76266e3e6d8ab3e3c8870a53ed9526ec0a2c81c0ddbfae8728e50d32
Oct 30 17:27:00.757881 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.757850    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:27:00.757951 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.757919    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:27:00.757986 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.757948    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:27:00.839167 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.839134    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b" event={"ID":"61e37504-76d8-47bf-8fa8-1a66d794e465","Type":"ContainerStarted","Data":"f27f2c87a1283ab0a1164f396e2868bdf2045780b19a1060ccae592e222ca276"}
Oct 30 17:27:00.839385 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.839190    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b" event={"ID":"61e37504-76d8-47bf-8fa8-1a66d794e465","Type":"ContainerStarted","Data":"4965466a76266e3e6d8ab3e3c8870a53ed9526ec0a2c81c0ddbfae8728e50d32"}
Oct 30 17:27:00.935579 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:00.935513    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b" podStartSLOduration=0.935495699 podStartE2EDuration="935.495699ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:00.930991698 +0000 UTC m=+2397.365893925" watchObservedRunningTime="2026-10-30 17:27:00.935495699 +0000 UTC m=+2397.370397904"
Oct 30 17:27:01.409528 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:01.409191    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"
Oct 30 17:27:01.410978 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:01.410941    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 17:27:01.411121 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:01.411033    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b" podUID="61e37504-76d8-47bf-8fa8-1a66d794e465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 17:27:02.408314 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:02.408271    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 17:27:02.408549 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:02.408329    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b" podUID="61e37504-76d8-47bf-8fa8-1a66d794e465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 17:27:03.408912 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:03.408871    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 17:27:03.409337 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:03.408972    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b" podUID="61e37504-76d8-47bf-8fa8-1a66d794e465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 17:27:04.408350 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:04.408313    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 17:27:04.408575 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:04.408383    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b" podUID="61e37504-76d8-47bf-8fa8-1a66d794e465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 17:27:05.114709 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:05.114670    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:27:05.408459 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:05.408354    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 17:27:05.408459 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:05.408448    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b" podUID="61e37504-76d8-47bf-8fa8-1a66d794e465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 17:27:06.413427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:06.412748    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 17:27:06.413427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:06.412834    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b" podUID="61e37504-76d8-47bf-8fa8-1a66d794e465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 17:27:07.413339 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:07.413304    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 17:27:07.413652 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:07.413620    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b" podUID="61e37504-76d8-47bf-8fa8-1a66d794e465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 17:27:08.039425 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:08.025503    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:27:08.416775 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:08.416688    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 17:27:08.417335 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:08.417305    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b" podUID="61e37504-76d8-47bf-8fa8-1a66d794e465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 17:27:09.414886 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:09.414855    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"
Oct 30 17:27:09.415303 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:09.415282    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"
Oct 30 17:27:09.416119 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:09.416101    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"
Oct 30 17:27:26.980155 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:27:26.980113    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:27:27.360163 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:27.360063    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns"]
Oct 30 17:27:27.366261 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:27.366237    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns"
Oct 30 17:27:27.381697 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:27.381666    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-b87f3c-serving-cert\""
Oct 30 17:27:27.401487 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:27.401455    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns"]
Oct 30 17:27:27.506121 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:27.506081    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrldx\" (UniqueName: \"kubernetes.io/projected/73875344-e1af-4208-86a8-681a05925c66-kube-api-access-vrldx\") pod \"maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns\" (UID: \"73875344-e1af-4208-86a8-681a05925c66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns"
Oct 30 17:27:27.506303 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:27.506234    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73875344-e1af-4208-86a8-681a05925c66-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns\" (UID: \"73875344-e1af-4208-86a8-681a05925c66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns"
Oct 30 17:27:27.506442 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:27.506390    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73875344-e1af-4208-86a8-681a05925c66-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns\" (UID: \"73875344-e1af-4208-86a8-681a05925c66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns"
Oct 30 17:27:27.506508 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:27.506493    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73875344-e1af-4208-86a8-681a05925c66-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns\" (UID: \"73875344-e1af-4208-86a8-681a05925c66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns"
Oct 30 17:27:27.608575 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:27.608533    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73875344-e1af-4208-86a8-681a05925c66-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns\" (UID: \"73875344-e1af-4208-86a8-681a05925c66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns"
Oct 30 17:27:27.608786 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:27.608629    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73875344-e1af-4208-86a8-681a05925c66-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns\" (UID: \"73875344-e1af-4208-86a8-681a05925c66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns"
Oct 30 17:27:27.608786 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:27.608724    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrldx\" (UniqueName: \"kubernetes.io/projected/73875344-e1af-4208-86a8-681a05925c66-kube-api-access-vrldx\") pod \"maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns\" (UID: \"73875344-e1af-4208-86a8-681a05925c66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns"
Oct 30 17:27:27.608933 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:27.608873    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73875344-e1af-4208-86a8-681a05925c66-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns\" (UID: \"73875344-e1af-4208-86a8-681a05925c66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns"
Oct 30 17:27:27.609187 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:27:27.609166    4433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b87f3c-serving-cert: secret "maas-api-e2e-shared-a-w6-b87f3c-serving-cert" not found
Oct 30 17:27:27.609279 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:27:27.609251    4433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73875344-e1af-4208-86a8-681a05925c66-maas-api-tls podName:73875344-e1af-4208-86a8-681a05925c66 nodeName:}" failed. No retries permitted until 2026-10-30 17:27:28.109230739 +0000 UTC m=+2424.544132925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/73875344-e1af-4208-86a8-681a05925c66-maas-api-tls") pod "maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns" (UID: "73875344-e1af-4208-86a8-681a05925c66") : secret "maas-api-e2e-shared-a-w6-b87f3c-serving-cert" not found
Oct 30 17:27:27.610222 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:27.610156    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73875344-e1af-4208-86a8-681a05925c66-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns\" (UID: \"73875344-e1af-4208-86a8-681a05925c66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns"
Oct 30 17:27:27.612661 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:27.612636    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73875344-e1af-4208-86a8-681a05925c66-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns\" (UID: \"73875344-e1af-4208-86a8-681a05925c66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns"
Oct 30 17:27:27.635866 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:27.635825    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrldx\" (UniqueName: \"kubernetes.io/projected/73875344-e1af-4208-86a8-681a05925c66-kube-api-access-vrldx\") pod \"maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns\" (UID: \"73875344-e1af-4208-86a8-681a05925c66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns"
Oct 30 17:27:28.116933 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:28.116895    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73875344-e1af-4208-86a8-681a05925c66-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns\" (UID: \"73875344-e1af-4208-86a8-681a05925c66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns"
Oct 30 17:27:28.119452 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:28.119427    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73875344-e1af-4208-86a8-681a05925c66-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns\" (UID: \"73875344-e1af-4208-86a8-681a05925c66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns"
Oct 30 17:27:28.280655 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:28.280620    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns"
Oct 30 17:27:28.597381 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:28.597356    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns"]
Oct 30 17:27:28.599587 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:27:28.599556    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73875344_e1af_4208_86a8_681a05925c66.slice/crio-9364b82f83344ed1b60ba329b92f0f31245b333fdb5db394bb479dc9c558364c WatchSource:0}: Error finding container 9364b82f83344ed1b60ba329b92f0f31245b333fdb5db394bb479dc9c558364c: Status 404 returned error can't find the container with id 9364b82f83344ed1b60ba329b92f0f31245b333fdb5db394bb479dc9c558364c
Oct 30 17:27:29.316934 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:29.316895    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns" event={"ID":"73875344-e1af-4208-86a8-681a05925c66","Type":"ContainerStarted","Data":"97803310540c97573daf259b93926a3f8d15c2d8276a4be2a06dd69b3e6fdec2"}
Oct 30 17:27:29.317381 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:29.316957    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns" event={"ID":"73875344-e1af-4208-86a8-681a05925c66","Type":"ContainerStarted","Data":"9364b82f83344ed1b60ba329b92f0f31245b333fdb5db394bb479dc9c558364c"}
Oct 30 17:27:29.317381 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:29.316992    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns"
Oct 30 17:27:29.361020 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:29.360955    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns" podStartSLOduration=2.053494753 podStartE2EDuration="2.360937936s" totalImagesPullingTime="307.443183ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:29.358925658 +0000 UTC m=+2425.793827878" watchObservedRunningTime="2026-10-30 17:27:29.360937936 +0000 UTC m=+2425.795840141"
Oct 30 17:27:32.398086 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:32.398049    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"]
Oct 30 17:27:32.402132 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:32.402116    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"
Oct 30 17:27:32.433148 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:32.433121    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-6d4f99-openshift-default-dockercfg-chllj\""
Oct 30 17:27:32.440438 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:32.437142    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"]
Oct 30 17:27:32.485505 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:32.485464    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1edce449-0010-4d41-908e-db02f1c508f9-workload-certs\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"
Oct 30 17:27:32.485505 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:32.485512    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1edce449-0010-4d41-908e-db02f1c508f9-istio-podinfo\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"
Oct 30 17:27:32.485731 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:32.485541    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1edce449-0010-4d41-908e-db02f1c508f9-istio-token\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"
Oct 30 17:27:32.485731 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:32.485667    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1edce449-0010-4d41-908e-db02f1c508f9-istio-data\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"
Oct 30 17:27:32.485731 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:32.485711    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntx88\" (UniqueName: \"kubernetes.io/projected/1edce449-0010-4d41-908e-db02f1c508f9-kube-api-access-ntx88\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"
Oct 30 17:27:32.485868 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:32.485770    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1edce449-0010-4d41-908e-db02f1c508f9-workload-socket\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"
Oct 30 17:27:32.485868 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:32.485791    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1edce449-0010-4d41-908e-db02f1c508f9-credential-socket\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"
Oct 30 17:27:32.485977 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:32.485902    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1edce449-0010-4d41-908e-db02f1c508f9-istiod-ca-cert\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"
Oct 30 17:27:32.486061 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:32.486039    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1edce449-0010-4d41-908e-db02f1c508f9-istio-envoy\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"
Oct 30 17:27:32.587239 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:32.587196    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1edce449-0010-4d41-908e-db02f1c508f9-istio-envoy\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"
Oct 30 17:27:32.587462 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:32.587302    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1edce449-0010-4d41-908e-db02f1c508f9-workload-certs\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"
Oct 30 17:27:32.587462 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:32.587340    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1edce449-0010-4d41-908e-db02f1c508f9-istio-podinfo\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"
Oct 30 17:27:32.587462 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:32.587380    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1edce449-0010-4d41-908e-db02f1c508f9-istio-token\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"
Oct 30 17:27:32.587462 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:32.587430    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1edce449-0010-4d41-908e-db02f1c508f9-istio-data\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"
Oct 30 17:27:32.587730 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:32.587600    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntx88\" (UniqueName: \"kubernetes.io/projected/1edce449-0010-4d41-908e-db02f1c508f9-kube-api-access-ntx88\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"
Oct 30 17:27:32.587823 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:32.587799    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1edce449-0010-4d41-908e-db02f1c508f9-istio-data\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"
Oct 30 17:27:32.587896 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:32.587825    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1edce449-0010-4d41-908e-db02f1c508f9-workload-certs\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"
Oct 30 17:27:32.588060 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:32.588037    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1edce449-0010-4d41-908e-db02f1c508f9-workload-socket\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"
Oct 30 17:27:32.588124 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:32.588071    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1edce449-0010-4d41-908e-db02f1c508f9-credential-socket\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"
Oct 30 17:27:32.588181 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:32.588145    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1edce449-0010-4d41-908e-db02f1c508f9-istiod-ca-cert\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"
Oct 30 17:27:32.588390 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:32.588369    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1edce449-0010-4d41-908e-db02f1c508f9-workload-socket\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"
Oct 30 17:27:32.588495 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:32.588465    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1edce449-0010-4d41-908e-db02f1c508f9-credential-socket\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"
Oct 30 17:27:32.588912 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:32.588892    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1edce449-0010-4d41-908e-db02f1c508f9-istiod-ca-cert\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"
Oct 30 17:27:32.590185 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:32.590156    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1edce449-0010-4d41-908e-db02f1c508f9-istio-envoy\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"
Oct 30 17:27:32.590468 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:32.590447    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1edce449-0010-4d41-908e-db02f1c508f9-istio-podinfo\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"
Oct 30 17:27:32.610524 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:32.610495    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1edce449-0010-4d41-908e-db02f1c508f9-istio-token\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"
Oct 30 17:27:32.610871 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:32.610853    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntx88\" (UniqueName: \"kubernetes.io/projected/1edce449-0010-4d41-908e-db02f1c508f9-kube-api-access-ntx88\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"
Oct 30 17:27:32.716018 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:32.715976    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"
Oct 30 17:27:33.036589 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:33.036563    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"]
Oct 30 17:27:33.038391 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:27:33.038368    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1edce449_0010_4d41_908e_db02f1c508f9.slice/crio-ccf58336975ec8cdeb730406fbf0470229aab6f570810c05e84d912c30d25176 WatchSource:0}: Error finding container ccf58336975ec8cdeb730406fbf0470229aab6f570810c05e84d912c30d25176: Status 404 returned error can't find the container with id ccf58336975ec8cdeb730406fbf0470229aab6f570810c05e84d912c30d25176
Oct 30 17:27:33.041450 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:33.041378    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:27:33.041542 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:33.041502    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:27:33.041542 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:33.041534    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:27:33.393248 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:33.393152    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh" event={"ID":"1edce449-0010-4d41-908e-db02f1c508f9","Type":"ContainerStarted","Data":"801a4a946d4fd753424aaaee540ca2cacb5841d9e397e04d5d99de630eef6e5a"}
Oct 30 17:27:33.393248 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:33.393210    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh" event={"ID":"1edce449-0010-4d41-908e-db02f1c508f9","Type":"ContainerStarted","Data":"ccf58336975ec8cdeb730406fbf0470229aab6f570810c05e84d912c30d25176"}
Oct 30 17:27:33.440970 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:33.440916    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh" podStartSLOduration=1.440902471 podStartE2EDuration="1.440902471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:33.437211916 +0000 UTC m=+2429.872114156" watchObservedRunningTime="2026-10-30 17:27:33.440902471 +0000 UTC m=+2429.875804677"
Oct 30 17:27:33.716460 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:33.716429    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"
Oct 30 17:27:34.718602 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:34.718564    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 17:27:34.719001 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:34.718634    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh" podUID="1edce449-0010-4d41-908e-db02f1c508f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 17:27:35.447766 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:35.446593    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns"
Oct 30 17:27:35.718182 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:35.718091    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 17:27:35.718532 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:35.718494    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh" podUID="1edce449-0010-4d41-908e-db02f1c508f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 17:27:36.718428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:36.717702    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 17:27:36.718428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:36.717779    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh" podUID="1edce449-0010-4d41-908e-db02f1c508f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 17:27:37.716807 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:37.716774    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 17:27:37.716987 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:37.716827    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh" podUID="1edce449-0010-4d41-908e-db02f1c508f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 17:27:38.716591 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:38.716553    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 17:27:38.716982 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:38.716608    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh" podUID="1edce449-0010-4d41-908e-db02f1c508f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 17:27:39.716914 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:39.716881    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 17:27:39.717330 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:39.716947    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh" podUID="1edce449-0010-4d41-908e-db02f1c508f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 17:27:40.721312 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:40.721278    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"
Oct 30 17:27:40.721841 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:40.721693    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"
Oct 30 17:27:40.722658 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:40.722632    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"
Oct 30 17:27:51.739669 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:51.739625    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"]
Oct 30 17:27:51.744482 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:51.744453    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"
Oct 30 17:27:51.762015 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:51.761985    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-8681b5-openshift-default-dockercfg-pd5qg\""
Oct 30 17:27:51.778715 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:51.778678    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"]
Oct 30 17:27:51.823163 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:51.823082    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjzpz\" (UniqueName: \"kubernetes.io/projected/303728db-2515-4854-a0ca-e8726a6414f5-kube-api-access-cjzpz\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"
Oct 30 17:27:51.823363 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:51.823259    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/303728db-2515-4854-a0ca-e8726a6414f5-workload-socket\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"
Oct 30 17:27:51.823488 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:51.823383    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/303728db-2515-4854-a0ca-e8726a6414f5-istio-podinfo\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"
Oct 30 17:27:51.823488 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:51.823443    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/303728db-2515-4854-a0ca-e8726a6414f5-istio-token\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"
Oct 30 17:27:51.823614 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:51.823537    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/303728db-2515-4854-a0ca-e8726a6414f5-credential-socket\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"
Oct 30 17:27:51.823674 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:51.823620    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/303728db-2515-4854-a0ca-e8726a6414f5-istio-data\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"
Oct 30 17:27:51.823723 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:51.823670    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/303728db-2515-4854-a0ca-e8726a6414f5-istio-envoy\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"
Oct 30 17:27:51.824315 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:51.824291    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/303728db-2515-4854-a0ca-e8726a6414f5-workload-certs\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"
Oct 30 17:27:51.824477 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:51.824327    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/303728db-2515-4854-a0ca-e8726a6414f5-istiod-ca-cert\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"
Oct 30 17:27:51.925658 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:51.925612    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/303728db-2515-4854-a0ca-e8726a6414f5-istio-envoy\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"
Oct 30 17:27:51.925884 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:51.925866    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/303728db-2515-4854-a0ca-e8726a6414f5-workload-certs\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"
Oct 30 17:27:51.925961 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:51.925906    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/303728db-2515-4854-a0ca-e8726a6414f5-istiod-ca-cert\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"
Oct 30 17:27:51.926020 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:51.925974    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjzpz\" (UniqueName: \"kubernetes.io/projected/303728db-2515-4854-a0ca-e8726a6414f5-kube-api-access-cjzpz\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"
Oct 30 17:27:51.926076 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:51.926037    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/303728db-2515-4854-a0ca-e8726a6414f5-workload-socket\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"
Oct 30 17:27:51.926076 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:51.926065    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/303728db-2515-4854-a0ca-e8726a6414f5-istio-podinfo\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"
Oct 30 17:27:51.926192 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:51.926086    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/303728db-2515-4854-a0ca-e8726a6414f5-istio-token\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"
Oct 30 17:27:51.926192 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:51.926115    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/303728db-2515-4854-a0ca-e8726a6414f5-credential-socket\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"
Oct 30 17:27:51.926192 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:51.926150    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/303728db-2515-4854-a0ca-e8726a6414f5-istio-data\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"
Oct 30 17:27:51.926426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:51.926379    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/303728db-2515-4854-a0ca-e8726a6414f5-workload-certs\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"
Oct 30 17:27:51.927117 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:51.926782    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/303728db-2515-4854-a0ca-e8726a6414f5-istiod-ca-cert\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"
Oct 30 17:27:51.927117 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:51.927032    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/303728db-2515-4854-a0ca-e8726a6414f5-istio-data\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"
Oct 30 17:27:51.927395 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:51.927049    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/303728db-2515-4854-a0ca-e8726a6414f5-credential-socket\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"
Oct 30 17:27:51.927910 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:51.927869    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/303728db-2515-4854-a0ca-e8726a6414f5-workload-socket\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"
Oct 30 17:27:51.964669 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:51.930868    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/303728db-2515-4854-a0ca-e8726a6414f5-istio-podinfo\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"
Oct 30 17:27:51.964669 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:51.963194    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/303728db-2515-4854-a0ca-e8726a6414f5-istio-token\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"
Oct 30 17:27:51.969596 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:51.968537    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/303728db-2515-4854-a0ca-e8726a6414f5-istio-envoy\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"
Oct 30 17:27:51.969596 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:51.969347    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjzpz\" (UniqueName: \"kubernetes.io/projected/303728db-2515-4854-a0ca-e8726a6414f5-kube-api-access-cjzpz\") pod \"e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") " pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"
Oct 30 17:27:52.060879 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:52.060795    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"
Oct 30 17:27:52.382308 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:52.382279    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"]
Oct 30 17:27:52.384228 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:27:52.384189    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303728db_2515_4854_a0ca_e8726a6414f5.slice/crio-3aa94546cce0276d07adfe37e9ba5d1c1f98a0999a5c6bfa76478f9fe9c4c220 WatchSource:0}: Error finding container 3aa94546cce0276d07adfe37e9ba5d1c1f98a0999a5c6bfa76478f9fe9c4c220: Status 404 returned error can't find the container with id 3aa94546cce0276d07adfe37e9ba5d1c1f98a0999a5c6bfa76478f9fe9c4c220
Oct 30 17:27:52.387169 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:52.387128    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:27:52.387277 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:52.387200    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:27:52.387277 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:52.387232    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:27:52.921426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:52.915857    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg" event={"ID":"303728db-2515-4854-a0ca-e8726a6414f5","Type":"ContainerStarted","Data":"8f97e87ff243fa618bcea3b25297bd69d3ac3c710e362f714b1b5637249c17c3"}
Oct 30 17:27:52.921426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:52.915938    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg" event={"ID":"303728db-2515-4854-a0ca-e8726a6414f5","Type":"ContainerStarted","Data":"3aa94546cce0276d07adfe37e9ba5d1c1f98a0999a5c6bfa76478f9fe9c4c220"}
Oct 30 17:27:53.060738 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:53.060658    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg" podStartSLOduration=2.060642672 podStartE2EDuration="2.060642672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:53.046829653 +0000 UTC m=+2449.481731859" watchObservedRunningTime="2026-10-30 17:27:53.060642672 +0000 UTC m=+2449.495544878"
Oct 30 17:27:53.063544 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:53.063495    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 17:27:53.063801 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:53.063555    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg" podUID="303728db-2515-4854-a0ca-e8726a6414f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 17:27:53.064149 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:53.064119    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"
Oct 30 17:27:54.065453 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:54.063863    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 17:27:54.065453 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:54.063952    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg" podUID="303728db-2515-4854-a0ca-e8726a6414f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 17:27:55.065916 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:55.065862    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 17:27:55.066540 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:55.066503    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg" podUID="303728db-2515-4854-a0ca-e8726a6414f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 17:27:56.068563 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:56.068520    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 17:27:56.069165 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:56.069128    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg" podUID="303728db-2515-4854-a0ca-e8726a6414f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 17:27:57.018482 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:27:57.018441    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:27:57.068472 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:57.063332    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 17:27:57.068472 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:57.063395    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg" podUID="303728db-2515-4854-a0ca-e8726a6414f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 17:27:58.077429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:58.070794    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 17:27:58.077429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:58.070855    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg" podUID="303728db-2515-4854-a0ca-e8726a6414f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 17:27:59.064703 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:59.064660    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 17:27:59.064918 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:27:59.064726    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg" podUID="303728db-2515-4854-a0ca-e8726a6414f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 17:28:00.066736 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:00.066700    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"
Oct 30 17:28:00.067121 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:00.067063    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"
Oct 30 17:28:00.067739 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:00.067720    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"
Oct 30 17:28:17.154523 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:17.154297    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:28:23.282429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:23.270046    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"]
Oct 30 17:28:23.289991 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:23.289952    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"
Oct 30 17:28:23.322462 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:23.318756    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-41cb25-openshift-default-dockercfg-wsmgg\""
Oct 30 17:28:23.322462 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:23.321545    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"]
Oct 30 17:28:23.346492 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:23.346457    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr9g5\" (UniqueName: \"kubernetes.io/projected/988a3f4c-5f27-474e-a55a-38f5c555d374-kube-api-access-rr9g5\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"
Oct 30 17:28:23.346803 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:23.346771    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/988a3f4c-5f27-474e-a55a-38f5c555d374-workload-certs\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"
Oct 30 17:28:23.347114 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:23.346853    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/988a3f4c-5f27-474e-a55a-38f5c555d374-istio-data\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"
Oct 30 17:28:23.347114 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:23.346943    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/988a3f4c-5f27-474e-a55a-38f5c555d374-istiod-ca-cert\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"
Oct 30 17:28:23.347114 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:23.347042    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/988a3f4c-5f27-474e-a55a-38f5c555d374-workload-socket\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"
Oct 30 17:28:23.347114 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:23.347095    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/988a3f4c-5f27-474e-a55a-38f5c555d374-istio-podinfo\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"
Oct 30 17:28:23.347513 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:23.347438    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/988a3f4c-5f27-474e-a55a-38f5c555d374-credential-socket\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"
Oct 30 17:28:23.347571 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:23.347513    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/988a3f4c-5f27-474e-a55a-38f5c555d374-istio-envoy\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"
Oct 30 17:28:23.347629 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:23.347572    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/988a3f4c-5f27-474e-a55a-38f5c555d374-istio-token\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"
Oct 30 17:28:23.448605 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:23.448563    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/988a3f4c-5f27-474e-a55a-38f5c555d374-workload-socket\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"
Oct 30 17:28:23.448809 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:23.448629    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/988a3f4c-5f27-474e-a55a-38f5c555d374-istio-podinfo\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"
Oct 30 17:28:23.448809 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:23.448655    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/988a3f4c-5f27-474e-a55a-38f5c555d374-credential-socket\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"
Oct 30 17:28:23.448809 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:23.448699    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/988a3f4c-5f27-474e-a55a-38f5c555d374-istio-envoy\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"
Oct 30 17:28:23.448809 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:23.448751    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/988a3f4c-5f27-474e-a55a-38f5c555d374-istio-token\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"
Oct 30 17:28:23.448809 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:23.448803    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr9g5\" (UniqueName: \"kubernetes.io/projected/988a3f4c-5f27-474e-a55a-38f5c555d374-kube-api-access-rr9g5\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"
Oct 30 17:28:23.449093 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:23.448869    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/988a3f4c-5f27-474e-a55a-38f5c555d374-workload-certs\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"
Oct 30 17:28:23.449093 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:23.448932    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/988a3f4c-5f27-474e-a55a-38f5c555d374-istio-data\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"
Oct 30 17:28:23.449093 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:23.449023    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/988a3f4c-5f27-474e-a55a-38f5c555d374-istiod-ca-cert\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"
Oct 30 17:28:23.449271 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:23.449096    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/988a3f4c-5f27-474e-a55a-38f5c555d374-credential-socket\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"
Oct 30 17:28:23.449376 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:23.449353    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/988a3f4c-5f27-474e-a55a-38f5c555d374-istio-data\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"
Oct 30 17:28:23.449493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:23.449471    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/988a3f4c-5f27-474e-a55a-38f5c555d374-workload-socket\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"
Oct 30 17:28:23.449731 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:23.449708    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/988a3f4c-5f27-474e-a55a-38f5c555d374-workload-certs\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"
Oct 30 17:28:23.449868 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:23.449846    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/988a3f4c-5f27-474e-a55a-38f5c555d374-istiod-ca-cert\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"
Oct 30 17:28:23.487482 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:23.479007    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/988a3f4c-5f27-474e-a55a-38f5c555d374-istio-podinfo\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"
Oct 30 17:28:23.487482 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:23.479224    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/988a3f4c-5f27-474e-a55a-38f5c555d374-istio-envoy\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"
Oct 30 17:28:23.487482 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:23.486975    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/988a3f4c-5f27-474e-a55a-38f5c555d374-istio-token\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"
Oct 30 17:28:23.489031 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:23.489001    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr9g5\" (UniqueName: \"kubernetes.io/projected/988a3f4c-5f27-474e-a55a-38f5c555d374-kube-api-access-rr9g5\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"
Oct 30 17:28:23.605641 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:23.605553    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"
Oct 30 17:28:24.026054 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:24.019521    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"]
Oct 30 17:28:24.050549 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:24.049698    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:28:24.050549 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:24.049828    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:28:24.050549 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:24.049881    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:28:24.332799 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:24.332724    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p" event={"ID":"988a3f4c-5f27-474e-a55a-38f5c555d374","Type":"ContainerStarted","Data":"249cafe4120350287f8f7e2098d7c02d31445a7d89828b3832f1bf0fe33888c7"}
Oct 30 17:28:25.341014 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:25.340977    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p" event={"ID":"988a3f4c-5f27-474e-a55a-38f5c555d374","Type":"ContainerStarted","Data":"7d0a57f4cd1b6a8c2b987ba8ac99ffa196721f72a65ce1fdc9b0237f5e5286f8"}
Oct 30 17:28:25.410442 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:25.410357    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p" podStartSLOduration=2.410337453 podStartE2EDuration="2.410337453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:28:25.409192727 +0000 UTC m=+2481.844094934" watchObservedRunningTime="2026-10-30 17:28:25.410337453 +0000 UTC m=+2481.845239651"
Oct 30 17:28:25.607818 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:25.607723    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"
Oct 30 17:28:25.608003 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:25.607884    4433 patch_prober.go:29] interesting pod/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 17:28:25.608003 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:25.607962    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p" podUID="988a3f4c-5f27-474e-a55a-38f5c555d374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 17:28:26.606506 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:26.606422    4433 patch_prober.go:29] interesting pod/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 17:28:26.606506 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:26.606481    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p" podUID="988a3f4c-5f27-474e-a55a-38f5c555d374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 17:28:27.008240 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:28:27.008204    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:28:27.613579 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:27.609793    4433 patch_prober.go:29] interesting pod/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 17:28:27.613579 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:27.609853    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p" podUID="988a3f4c-5f27-474e-a55a-38f5c555d374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 17:28:28.606641 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:28.606603    4433 patch_prober.go:29] interesting pod/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 17:28:28.606902 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:28.606658    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p" podUID="988a3f4c-5f27-474e-a55a-38f5c555d374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 17:28:29.606287 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:29.606253    4433 patch_prober.go:29] interesting pod/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 17:28:29.606766 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:29.606306    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p" podUID="988a3f4c-5f27-474e-a55a-38f5c555d374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 17:28:29.876524 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:29.869531    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:28:29.876524 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:29.869616    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb" podUID="aefefaea-deba-44ec-a357-4fb4ba5e7f6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:28:30.606189 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:30.606149    4433 patch_prober.go:29] interesting pod/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 17:28:30.606368 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:30.606217    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p" podUID="988a3f4c-5f27-474e-a55a-38f5c555d374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 17:28:31.614823 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:31.614796    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"
Oct 30 17:28:31.615345 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:31.615322    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"
Oct 30 17:28:31.615997 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:31.615973    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"
Oct 30 17:28:32.413553 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:32.413519    4433 generic.go:356] "Generic (PLEG): container finished" podID="aefefaea-deba-44ec-a357-4fb4ba5e7f6e" containerID="284070c3fc1cda348dc8dd123a9fd8793da2aea27df2249862097a86b81ea93d" exitCode=0
Oct 30 17:28:32.413772 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:32.413600    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb" event={"ID":"aefefaea-deba-44ec-a357-4fb4ba5e7f6e","Type":"ContainerDied","Data":"284070c3fc1cda348dc8dd123a9fd8793da2aea27df2249862097a86b81ea93d"}
Oct 30 17:28:32.414193 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:32.414175    4433 scope.go:122] "RemoveContainer" containerID="284070c3fc1cda348dc8dd123a9fd8793da2aea27df2249862097a86b81ea93d"
Oct 30 17:28:32.415925 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:32.415891    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:28:32.415997 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:32.415958    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:28:32.415997 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:32.415989    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:28:33.421049 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:33.421011    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb" event={"ID":"aefefaea-deba-44ec-a357-4fb4ba5e7f6e","Type":"ContainerStarted","Data":"a424e1c49acb61cdd55378c196749cb7cc6cd98c3aa808c67d9cf4ba91ae229a"}
Oct 30 17:28:34.127933 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:34.127898    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:28:34.128289 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:34.128265    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 17:28:34.128487 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:34.128459    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb" podUID="aefefaea-deba-44ec-a357-4fb4ba5e7f6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 17:28:35.122451 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:35.122417    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 17:28:35.122864 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:35.122470    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb" podUID="aefefaea-deba-44ec-a357-4fb4ba5e7f6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 17:28:36.121485 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:36.121448    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 17:28:36.121672 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:36.121515    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb" podUID="aefefaea-deba-44ec-a357-4fb4ba5e7f6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 17:28:37.121673 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:37.121637    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 17:28:37.122185 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:37.121708    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb" podUID="aefefaea-deba-44ec-a357-4fb4ba5e7f6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 17:28:38.122924 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:38.122882    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 17:28:38.123471 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:38.122966    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb" podUID="aefefaea-deba-44ec-a357-4fb4ba5e7f6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 17:28:39.337437 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:39.337389    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:28:39.339325 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:39.339300    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:28:39.351450 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:39.351396    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:28:44.976216 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:44.976180    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk"]
Oct 30 17:28:44.981168 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:44.981138    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk"
Oct 30 17:28:45.008226 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.008194    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-469ad7\"/\"kube-root-ca.crt\""
Oct 30 17:28:45.008429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.008333    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-469ad7\"/\"auth-test-model-469ad7-kserve-self-signed-certs\""
Oct 30 17:28:45.008429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.008421    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-469ad7\"/\"openshift-service-ca.crt\""
Oct 30 17:28:45.008554 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.008435    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-469ad7\"/\"default-dockercfg-6s6pz\""
Oct 30 17:28:45.013636 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.013604    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk"]
Oct 30 17:28:45.083944 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.083906    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d6ec843-3b36-407d-87bd-ae5a21472b61-dshm\") pod \"auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk\" (UID: \"8d6ec843-3b36-407d-87bd-ae5a21472b61\") " pod="ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk"
Oct 30 17:28:45.084163 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.084066    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d6ec843-3b36-407d-87bd-ae5a21472b61-home\") pod \"auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk\" (UID: \"8d6ec843-3b36-407d-87bd-ae5a21472b61\") " pod="ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk"
Oct 30 17:28:45.084163 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.084120    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jwwx\" (UniqueName: \"kubernetes.io/projected/8d6ec843-3b36-407d-87bd-ae5a21472b61-kube-api-access-5jwwx\") pod \"auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk\" (UID: \"8d6ec843-3b36-407d-87bd-ae5a21472b61\") " pod="ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk"
Oct 30 17:28:45.084292 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.084182    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d6ec843-3b36-407d-87bd-ae5a21472b61-model-cache\") pod \"auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk\" (UID: \"8d6ec843-3b36-407d-87bd-ae5a21472b61\") " pod="ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk"
Oct 30 17:28:45.084373 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.084350    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d6ec843-3b36-407d-87bd-ae5a21472b61-tmp-dir\") pod \"auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk\" (UID: \"8d6ec843-3b36-407d-87bd-ae5a21472b61\") " pod="ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk"
Oct 30 17:28:45.084480 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.084387    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d6ec843-3b36-407d-87bd-ae5a21472b61-tls-certs\") pod \"auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk\" (UID: \"8d6ec843-3b36-407d-87bd-ae5a21472b61\") " pod="ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk"
Oct 30 17:28:45.186125 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.186088    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d6ec843-3b36-407d-87bd-ae5a21472b61-dshm\") pod \"auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk\" (UID: \"8d6ec843-3b36-407d-87bd-ae5a21472b61\") " pod="ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk"
Oct 30 17:28:45.186348 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.186154    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d6ec843-3b36-407d-87bd-ae5a21472b61-home\") pod \"auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk\" (UID: \"8d6ec843-3b36-407d-87bd-ae5a21472b61\") " pod="ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk"
Oct 30 17:28:45.186348 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.186187    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jwwx\" (UniqueName: \"kubernetes.io/projected/8d6ec843-3b36-407d-87bd-ae5a21472b61-kube-api-access-5jwwx\") pod \"auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk\" (UID: \"8d6ec843-3b36-407d-87bd-ae5a21472b61\") " pod="ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk"
Oct 30 17:28:45.186348 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.186225    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d6ec843-3b36-407d-87bd-ae5a21472b61-model-cache\") pod \"auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk\" (UID: \"8d6ec843-3b36-407d-87bd-ae5a21472b61\") " pod="ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk"
Oct 30 17:28:45.186348 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.186312    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d6ec843-3b36-407d-87bd-ae5a21472b61-tmp-dir\") pod \"auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk\" (UID: \"8d6ec843-3b36-407d-87bd-ae5a21472b61\") " pod="ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk"
Oct 30 17:28:45.186348 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.186334    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d6ec843-3b36-407d-87bd-ae5a21472b61-tls-certs\") pod \"auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk\" (UID: \"8d6ec843-3b36-407d-87bd-ae5a21472b61\") " pod="ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk"
Oct 30 17:28:45.186676 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.186658    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d6ec843-3b36-407d-87bd-ae5a21472b61-home\") pod \"auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk\" (UID: \"8d6ec843-3b36-407d-87bd-ae5a21472b61\") " pod="ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk"
Oct 30 17:28:45.186772 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.186742    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d6ec843-3b36-407d-87bd-ae5a21472b61-model-cache\") pod \"auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk\" (UID: \"8d6ec843-3b36-407d-87bd-ae5a21472b61\") " pod="ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk"
Oct 30 17:28:45.187060 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.187031    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d6ec843-3b36-407d-87bd-ae5a21472b61-tmp-dir\") pod \"auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk\" (UID: \"8d6ec843-3b36-407d-87bd-ae5a21472b61\") " pod="ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk"
Oct 30 17:28:45.188810 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.188787    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d6ec843-3b36-407d-87bd-ae5a21472b61-dshm\") pod \"auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk\" (UID: \"8d6ec843-3b36-407d-87bd-ae5a21472b61\") " pod="ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk"
Oct 30 17:28:45.189469 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.189446    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d6ec843-3b36-407d-87bd-ae5a21472b61-tls-certs\") pod \"auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk\" (UID: \"8d6ec843-3b36-407d-87bd-ae5a21472b61\") " pod="ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk"
Oct 30 17:28:45.211424 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.211214    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jwwx\" (UniqueName: \"kubernetes.io/projected/8d6ec843-3b36-407d-87bd-ae5a21472b61-kube-api-access-5jwwx\") pod \"auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk\" (UID: \"8d6ec843-3b36-407d-87bd-ae5a21472b61\") " pod="ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk"
Oct 30 17:28:45.307487 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.300653    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk"
Oct 30 17:28:45.669904 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.669864    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk"]
Oct 30 17:28:45.671717 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:28:45.671687    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d6ec843_3b36_407d_87bd_ae5a21472b61.slice/crio-b3b89cba850bd86e96f10bddbcdda9b4bacecd298dc6eea4087883f35285c91d WatchSource:0}: Error finding container b3b89cba850bd86e96f10bddbcdda9b4bacecd298dc6eea4087883f35285c91d: Status 404 returned error can't find the container with id b3b89cba850bd86e96f10bddbcdda9b4bacecd298dc6eea4087883f35285c91d
Oct 30 17:28:45.705197 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.705163    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"]
Oct 30 17:28:45.713030 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.713006    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"
Oct 30 17:28:45.724179 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.724153    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"]
Oct 30 17:28:45.734604 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.734573    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-82243b-openshift-default-dockercfg-c4f5c\""
Oct 30 17:28:45.899980 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.899939    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-istio-envoy\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"
Oct 30 17:28:45.900138 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.900088    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-istio-token\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"
Oct 30 17:28:45.900182 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.900161    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-credential-socket\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"
Oct 30 17:28:45.900218 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.900198    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-istio-podinfo\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"
Oct 30 17:28:45.900295 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.900274    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-workload-certs\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"
Oct 30 17:28:45.900337 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.900313    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-istiod-ca-cert\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"
Oct 30 17:28:45.900392 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.900375    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq6p7\" (UniqueName: \"kubernetes.io/projected/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-kube-api-access-kq6p7\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"
Oct 30 17:28:45.900448 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.900433    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-workload-socket\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"
Oct 30 17:28:45.900596 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:45.900577    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-istio-data\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"
Oct 30 17:28:46.001726 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:46.001685    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-istio-token\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"
Oct 30 17:28:46.002222 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:46.001748    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-credential-socket\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"
Oct 30 17:28:46.002222 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:46.001818    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-istio-podinfo\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"
Oct 30 17:28:46.002222 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:46.001868    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-workload-certs\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"
Oct 30 17:28:46.002222 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:46.001895    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-istiod-ca-cert\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"
Oct 30 17:28:46.002222 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:46.001940    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq6p7\" (UniqueName: \"kubernetes.io/projected/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-kube-api-access-kq6p7\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"
Oct 30 17:28:46.002222 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:46.001981    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-workload-socket\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"
Oct 30 17:28:46.002222 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:46.002131    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-istio-data\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"
Oct 30 17:28:46.002222 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:46.002219    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-istio-envoy\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"
Oct 30 17:28:46.002698 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:46.002256    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-workload-certs\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"
Oct 30 17:28:46.002698 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:46.002304    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-credential-socket\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"
Oct 30 17:28:46.002698 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:46.002370    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-workload-socket\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"
Oct 30 17:28:46.002698 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:46.002498    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-istio-data\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"
Oct 30 17:28:46.002930 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:46.002910    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-istiod-ca-cert\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"
Oct 30 17:28:46.004397 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:46.004373    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-istio-envoy\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"
Oct 30 17:28:46.004506 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:46.004462    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-istio-podinfo\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"
Oct 30 17:28:46.027156 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:46.027118    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-istio-token\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"
Oct 30 17:28:46.027292 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:46.027237    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq6p7\" (UniqueName: \"kubernetes.io/projected/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-kube-api-access-kq6p7\") pod \"e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") " pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"
Oct 30 17:28:46.049440 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:46.049376    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"
Oct 30 17:28:46.425004 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:46.424978    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"]
Oct 30 17:28:46.426994 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:28:46.426967    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a7e2bc_f103_44c3_b5a4_4f93647a28e7.slice/crio-57abc099ccb4906eee0df7369d8138b44281157313ffa5ab8eb743072df4650d WatchSource:0}: Error finding container 57abc099ccb4906eee0df7369d8138b44281157313ffa5ab8eb743072df4650d: Status 404 returned error can't find the container with id 57abc099ccb4906eee0df7369d8138b44281157313ffa5ab8eb743072df4650d
Oct 30 17:28:46.430139 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:46.430102    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:28:46.430254 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:46.430180    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:28:46.430254 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:46.430214    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:28:46.565129 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:46.564528    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk" event={"ID":"8d6ec843-3b36-407d-87bd-ae5a21472b61","Type":"ContainerStarted","Data":"1511a123daec7817fbe00708c50d17b9b2a9bc57cfdc2784ceb8e054a3975e53"}
Oct 30 17:28:46.565129 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:46.564588    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk" event={"ID":"8d6ec843-3b36-407d-87bd-ae5a21472b61","Type":"ContainerStarted","Data":"b3b89cba850bd86e96f10bddbcdda9b4bacecd298dc6eea4087883f35285c91d"}
Oct 30 17:28:46.566880 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:46.566852    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s" event={"ID":"98a7e2bc-f103-44c3-b5a4-4f93647a28e7","Type":"ContainerStarted","Data":"21c04ed531636f8a1bcdc72e1f891223906b38224c65a1bdb8e8bf4325423aa9"}
Oct 30 17:28:46.567007 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:46.566894    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s" event={"ID":"98a7e2bc-f103-44c3-b5a4-4f93647a28e7","Type":"ContainerStarted","Data":"57abc099ccb4906eee0df7369d8138b44281157313ffa5ab8eb743072df4650d"}
Oct 30 17:28:46.608164 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:46.608102    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk" podStartSLOduration=2.60808448 podStartE2EDuration="2.60808448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:28:46.605324724 +0000 UTC m=+2503.040226930" watchObservedRunningTime="2026-10-30 17:28:46.60808448 +0000 UTC m=+2503.042986726"
Oct 30 17:28:46.652314 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:46.652258    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s" podStartSLOduration=1.652240515 podStartE2EDuration="1.652240515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:28:46.649040717 +0000 UTC m=+2503.083942924" watchObservedRunningTime="2026-10-30 17:28:46.652240515 +0000 UTC m=+2503.087142722"
Oct 30 17:28:47.050442 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:47.050383    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"
Oct 30 17:28:47.063624 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:47.063588    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 17:28:47.063819 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:47.063665    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s" podUID="98a7e2bc-f103-44c3-b5a4-4f93647a28e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 17:28:47.301367 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:47.301283    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk"
Oct 30 17:28:48.049808 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:48.049774    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 17:28:48.049978 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:48.049823    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s" podUID="98a7e2bc-f103-44c3-b5a4-4f93647a28e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 17:28:49.050369 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:49.050333    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 17:28:49.050912 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:49.050387    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s" podUID="98a7e2bc-f103-44c3-b5a4-4f93647a28e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 17:28:50.067440 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:50.064475    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 17:28:50.067440 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:50.064542    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s" podUID="98a7e2bc-f103-44c3-b5a4-4f93647a28e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 17:28:51.050862 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:51.050819    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 17:28:51.051153 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:51.051121    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s" podUID="98a7e2bc-f103-44c3-b5a4-4f93647a28e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 17:28:52.050211 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:52.050178    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 17:28:52.050746 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:52.050230    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s" podUID="98a7e2bc-f103-44c3-b5a4-4f93647a28e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 17:28:53.098813 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:53.098756    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"
Oct 30 17:28:53.099807 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:53.099785    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"
Oct 30 17:28:53.101443 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:53.101386    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"
Oct 30 17:28:55.301577 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:55.301542    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk"
Oct 30 17:28:55.314815 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:55.314787    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk"
Oct 30 17:28:55.789646 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:28:55.789612    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk"
Oct 30 17:28:56.979245 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:28:56.979207    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:29:06.585430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:06.564790    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:29:21.233129 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:21.233090    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-5b45f9975-lrcs9"]
Oct 30 17:29:21.239070 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:21.239046    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-5b45f9975-lrcs9"
Oct 30 17:29:21.257670 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:21.257644    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-93e020-dockercfg-p99rl\""
Oct 30 17:29:21.258906 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:21.258890    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-93e020\""
Oct 30 17:29:21.273650 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:21.273623    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-5b45f9975-lrcs9"]
Oct 30 17:29:21.425372 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:21.425331    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rk66\" (UniqueName: \"kubernetes.io/projected/3938c343-9631-413d-839a-76c566c23d3d-kube-api-access-8rk66\") pod \"payload-pre-processing-e2e-ipp-b-93e020-5b45f9975-lrcs9\" (UID: \"3938c343-9631-413d-839a-76c566c23d3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-5b45f9975-lrcs9"
Oct 30 17:29:21.425713 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:21.425687    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3938c343-9631-413d-839a-76c566c23d3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-93e020-5b45f9975-lrcs9\" (UID: \"3938c343-9631-413d-839a-76c566c23d3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-5b45f9975-lrcs9"
Oct 30 17:29:21.527194 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:21.527103    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rk66\" (UniqueName: \"kubernetes.io/projected/3938c343-9631-413d-839a-76c566c23d3d-kube-api-access-8rk66\") pod \"payload-pre-processing-e2e-ipp-b-93e020-5b45f9975-lrcs9\" (UID: \"3938c343-9631-413d-839a-76c566c23d3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-5b45f9975-lrcs9"
Oct 30 17:29:21.527363 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:21.527214    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3938c343-9631-413d-839a-76c566c23d3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-93e020-5b45f9975-lrcs9\" (UID: \"3938c343-9631-413d-839a-76c566c23d3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-5b45f9975-lrcs9"
Oct 30 17:29:21.532607 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:21.532560    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3938c343-9631-413d-839a-76c566c23d3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-93e020-5b45f9975-lrcs9\" (UID: \"3938c343-9631-413d-839a-76c566c23d3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-5b45f9975-lrcs9"
Oct 30 17:29:21.561036 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:21.561004    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rk66\" (UniqueName: \"kubernetes.io/projected/3938c343-9631-413d-839a-76c566c23d3d-kube-api-access-8rk66\") pod \"payload-pre-processing-e2e-ipp-b-93e020-5b45f9975-lrcs9\" (UID: \"3938c343-9631-413d-839a-76c566c23d3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-5b45f9975-lrcs9"
Oct 30 17:29:21.853777 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:21.853687    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-5b45f9975-lrcs9"
Oct 30 17:29:22.163438 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:22.163389    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-5b45f9975-lrcs9"]
Oct 30 17:29:22.165341 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:29:22.165312    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3938c343_9631_413d_839a_76c566c23d3d.slice/crio-65cd8fe468df8a79d117b1e0cbf7bb09cf1c5aa62f978f9f5db7c156105c7ecb WatchSource:0}: Error finding container 65cd8fe468df8a79d117b1e0cbf7bb09cf1c5aa62f978f9f5db7c156105c7ecb: Status 404 returned error can't find the container with id 65cd8fe468df8a79d117b1e0cbf7bb09cf1c5aa62f978f9f5db7c156105c7ecb
Oct 30 17:29:22.871679 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:22.871646    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-5b45f9975-lrcs9" event={"ID":"3938c343-9631-413d-839a-76c566c23d3d","Type":"ContainerStarted","Data":"9f58ad4abd7f1307f18c0b6615fc0fb5772f8cdecdc3ce6129223f76cc45ece9"}
Oct 30 17:29:22.872172 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:22.871700    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-5b45f9975-lrcs9" event={"ID":"3938c343-9631-413d-839a-76c566c23d3d","Type":"ContainerStarted","Data":"65cd8fe468df8a79d117b1e0cbf7bb09cf1c5aa62f978f9f5db7c156105c7ecb"}
Oct 30 17:29:22.872172 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:22.871761    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-5b45f9975-lrcs9"
Oct 30 17:29:22.912676 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:22.912623    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-5b45f9975-lrcs9" podStartSLOduration=1.912609142 podStartE2EDuration="1.912609142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:29:22.910824732 +0000 UTC m=+2539.345726962" watchObservedRunningTime="2026-10-30 17:29:22.912609142 +0000 UTC m=+2539.347511346"
Oct 30 17:29:26.977873 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:29:26.977833    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:29:33.954636 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:33.954608    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-5b45f9975-lrcs9"
Oct 30 17:29:56.978281 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:29:56.978196    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:29:58.781428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:58.776584    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"]
Oct 30 17:29:58.809434 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:58.808636    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:29:58.833427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:58.826732    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-3260f8-openshift-default-dockercfg-9zcr5\""
Oct 30 17:29:58.833427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:58.828421    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"]
Oct 30 17:29:58.896456 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:58.886325    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cd29788-1dd4-4f3a-910e-0c180daed229-istio-podinfo\") pod \"e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") " pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:29:58.896456 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:58.886381    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cd29788-1dd4-4f3a-910e-0c180daed229-workload-certs\") pod \"e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") " pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:29:58.896456 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:58.886440    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cd29788-1dd4-4f3a-910e-0c180daed229-istio-token\") pod \"e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") " pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:29:58.896456 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:58.886536    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd29788-1dd4-4f3a-910e-0c180daed229-workload-socket\") pod \"e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") " pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:29:58.896456 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:58.886583    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd29788-1dd4-4f3a-910e-0c180daed229-credential-socket\") pod \"e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") " pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:29:58.896456 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:58.886608    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrnf2\" (UniqueName: \"kubernetes.io/projected/0cd29788-1dd4-4f3a-910e-0c180daed229-kube-api-access-vrnf2\") pod \"e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") " pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:29:58.896456 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:58.886661    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cd29788-1dd4-4f3a-910e-0c180daed229-istio-data\") pod \"e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") " pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:29:58.896456 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:58.886752    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cd29788-1dd4-4f3a-910e-0c180daed229-istio-envoy\") pod \"e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") " pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:29:58.896456 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:58.886849    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cd29788-1dd4-4f3a-910e-0c180daed229-istiod-ca-cert\") pod \"e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") " pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:29:59.066620 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:59.060463    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cd29788-1dd4-4f3a-910e-0c180daed229-istio-podinfo\") pod \"e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") " pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:29:59.066620 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:59.060523    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cd29788-1dd4-4f3a-910e-0c180daed229-workload-certs\") pod \"e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") " pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:29:59.066620 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:59.060571    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cd29788-1dd4-4f3a-910e-0c180daed229-istio-token\") pod \"e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") " pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:29:59.066620 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:59.060629    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd29788-1dd4-4f3a-910e-0c180daed229-workload-socket\") pod \"e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") " pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:29:59.066620 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:59.060679    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd29788-1dd4-4f3a-910e-0c180daed229-credential-socket\") pod \"e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") " pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:29:59.066620 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:59.060705    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrnf2\" (UniqueName: \"kubernetes.io/projected/0cd29788-1dd4-4f3a-910e-0c180daed229-kube-api-access-vrnf2\") pod \"e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") " pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:29:59.066620 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:59.060765    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cd29788-1dd4-4f3a-910e-0c180daed229-istio-data\") pod \"e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") " pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:29:59.066620 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:59.060867    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cd29788-1dd4-4f3a-910e-0c180daed229-istio-envoy\") pod \"e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") " pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:29:59.066620 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:59.060970    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cd29788-1dd4-4f3a-910e-0c180daed229-istiod-ca-cert\") pod \"e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") " pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:29:59.066620 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:59.061827    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cd29788-1dd4-4f3a-910e-0c180daed229-istiod-ca-cert\") pod \"e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") " pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:29:59.066620 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:59.062167    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cd29788-1dd4-4f3a-910e-0c180daed229-workload-certs\") pod \"e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") " pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:29:59.066620 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:59.064129    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd29788-1dd4-4f3a-910e-0c180daed229-credential-socket\") pod \"e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") " pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:29:59.066620 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:59.064420    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cd29788-1dd4-4f3a-910e-0c180daed229-istio-data\") pod \"e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") " pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:29:59.066620 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:59.066523    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd29788-1dd4-4f3a-910e-0c180daed229-workload-socket\") pod \"e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") " pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:29:59.083914 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:59.083882    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cd29788-1dd4-4f3a-910e-0c180daed229-istio-envoy\") pod \"e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") " pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:29:59.091309 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:59.091246    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrnf2\" (UniqueName: \"kubernetes.io/projected/0cd29788-1dd4-4f3a-910e-0c180daed229-kube-api-access-vrnf2\") pod \"e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") " pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:29:59.092019 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:59.091956    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cd29788-1dd4-4f3a-910e-0c180daed229-istio-podinfo\") pod \"e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") " pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:29:59.093665 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:59.093607    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cd29788-1dd4-4f3a-910e-0c180daed229-istio-token\") pod \"e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") " pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:29:59.151928 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:59.151861    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:29:59.501831 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:59.501803    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"]
Oct 30 17:29:59.504017 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:29:59.503983    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cd29788_1dd4_4f3a_910e_0c180daed229.slice/crio-fb76cfdd30c9f41f6ead08b2c962f876b0d89d4385d5c94854490d9d29c04fe7 WatchSource:0}: Error finding container fb76cfdd30c9f41f6ead08b2c962f876b0d89d4385d5c94854490d9d29c04fe7: Status 404 returned error can't find the container with id fb76cfdd30c9f41f6ead08b2c962f876b0d89d4385d5c94854490d9d29c04fe7
Oct 30 17:29:59.507180 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:59.507143    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:29:59.507268 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:59.507215    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:29:59.507268 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:29:59.507244    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:30:00.158501 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.158461    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws"]
Oct 30 17:30:00.163952 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.163926    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws"
Oct 30 17:30:00.164615 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.164583    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf"]
Oct 30 17:30:00.168743 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.168722    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf"
Oct 30 17:30:00.181163 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.181141    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\""
Oct 30 17:30:00.192645 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.192617    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws"]
Oct 30 17:30:00.196980 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.196954    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889690-wzlh5"]
Oct 30 17:30:00.204806 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.201994    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf"]
Oct 30 17:30:00.204806 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.202102    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889690-wzlh5"
Oct 30 17:30:00.207462 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.206621    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj"]
Oct 30 17:30:00.212754 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.212360    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj"
Oct 30 17:30:00.251495 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.246503    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99"]
Oct 30 17:30:00.267430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.264771    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889690-wzlh5"]
Oct 30 17:30:00.267430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.264896    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99"
Oct 30 17:30:00.329630 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.329589    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3961d3d-0ada-44af-a530-2a7565e96257-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws\" (UID: \"c3961d3d-0ada-44af-a530-2a7565e96257\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws"
Oct 30 17:30:00.329630 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.329666    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x989w\" (UniqueName: \"kubernetes.io/projected/c3961d3d-0ada-44af-a530-2a7565e96257-kube-api-access-x989w\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws\" (UID: \"c3961d3d-0ada-44af-a530-2a7565e96257\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws"
Oct 30 17:30:00.330092 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.329821    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12e09dc3-bfd6-4b8b-90ca-8fab73d0c922-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf\" (UID: \"12e09dc3-bfd6-4b8b-90ca-8fab73d0c922\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf"
Oct 30 17:30:00.330092 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.329888    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqtp9\" (UniqueName: \"kubernetes.io/projected/12e09dc3-bfd6-4b8b-90ca-8fab73d0c922-kube-api-access-gqtp9\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf\" (UID: \"12e09dc3-bfd6-4b8b-90ca-8fab73d0c922\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf"
Oct 30 17:30:00.330092 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.329933    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt87z\" (UniqueName: \"kubernetes.io/projected/786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3-kube-api-access-gt87z\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889690-wzlh5\" (UID: \"786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889690-wzlh5"
Oct 30 17:30:00.330092 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.329972    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqs75\" (UniqueName: \"kubernetes.io/projected/73952f5d-729f-4433-aeeb-21c31b18c1d9-kube-api-access-dqs75\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99\" (UID: \"73952f5d-729f-4433-aeeb-21c31b18c1d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99"
Oct 30 17:30:00.330092 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.330033    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73952f5d-729f-4433-aeeb-21c31b18c1d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99\" (UID: \"73952f5d-729f-4433-aeeb-21c31b18c1d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99"
Oct 30 17:30:00.330092 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.330068    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbrcz\" (UniqueName: \"kubernetes.io/projected/5d225e1b-96bc-4632-8088-60a96b84999c-kube-api-access-nbrcz\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj\" (UID: \"5d225e1b-96bc-4632-8088-60a96b84999c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj"
Oct 30 17:30:00.330531 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.330119    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d225e1b-96bc-4632-8088-60a96b84999c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj\" (UID: \"5d225e1b-96bc-4632-8088-60a96b84999c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj"
Oct 30 17:30:00.330531 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.330166    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889690-wzlh5\" (UID: \"786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889690-wzlh5"
Oct 30 17:30:00.335017 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.333503    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch" event={"ID":"0cd29788-1dd4-4f3a-910e-0c180daed229","Type":"ContainerStarted","Data":"7c3a2a585b69006a59fa9b1fdcd7a0595f115e43433b23393e25ce2809c76c38"}
Oct 30 17:30:00.335017 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.333568    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch" event={"ID":"0cd29788-1dd4-4f3a-910e-0c180daed229","Type":"ContainerStarted","Data":"fb76cfdd30c9f41f6ead08b2c962f876b0d89d4385d5c94854490d9d29c04fe7"}
Oct 30 17:30:00.345815 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.341166    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n"]
Oct 30 17:30:00.346818 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.346795    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n"
Oct 30 17:30:00.349133 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.349110    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg"]
Oct 30 17:30:00.353254 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.353235    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj"]
Oct 30 17:30:00.353347 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.353316    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg"
Oct 30 17:30:00.356307 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.356275    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl"]
Oct 30 17:30:00.361089 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.361068    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99"]
Oct 30 17:30:00.361199 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.361151    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl"
Oct 30 17:30:00.380291 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.380266    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n"]
Oct 30 17:30:00.389040 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.389019    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889690-5pb4x"]
Oct 30 17:30:00.399594 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.399574    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-5pb4x"
Oct 30 17:30:00.407040 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.407018    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg"]
Oct 30 17:30:00.422894 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.422444    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl"]
Oct 30 17:30:00.423945 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.423843    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m"]
Oct 30 17:30:00.431534 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.431510    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l45sh\" (UniqueName: \"kubernetes.io/projected/f02fcdb6-55c4-4713-b726-ff1de3d3a7ba-kube-api-access-l45sh\") pod \"maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg\" (UID: \"f02fcdb6-55c4-4713-b726-ff1de3d3a7ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg"
Oct 30 17:30:00.431692 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.431672    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889690-wzlh5\" (UID: \"786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889690-wzlh5"
Oct 30 17:30:00.431891 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.431867    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/993f683e-25cd-4f6e-beff-4da2e7c6ea69-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889690-5pb4x\" (UID: \"993f683e-25cd-4f6e-beff-4da2e7c6ea69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-5pb4x"
Oct 30 17:30:00.432104 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.432084    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3961d3d-0ada-44af-a530-2a7565e96257-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws\" (UID: \"c3961d3d-0ada-44af-a530-2a7565e96257\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws"
Oct 30 17:30:00.432215 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.432197    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x989w\" (UniqueName: \"kubernetes.io/projected/c3961d3d-0ada-44af-a530-2a7565e96257-kube-api-access-x989w\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws\" (UID: \"c3961d3d-0ada-44af-a530-2a7565e96257\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws"
Oct 30 17:30:00.432309 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.432293    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cphvf\" (UniqueName: \"kubernetes.io/projected/993f683e-25cd-4f6e-beff-4da2e7c6ea69-kube-api-access-cphvf\") pod \"maas-api-key-cleanup-29889690-5pb4x\" (UID: \"993f683e-25cd-4f6e-beff-4da2e7c6ea69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-5pb4x"
Oct 30 17:30:00.432366 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.432328    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f02fcdb6-55c4-4713-b726-ff1de3d3a7ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg\" (UID: \"f02fcdb6-55c4-4713-b726-ff1de3d3a7ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg"
Oct 30 17:30:00.432923 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.432901    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3961d3d-0ada-44af-a530-2a7565e96257-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws\" (UID: \"c3961d3d-0ada-44af-a530-2a7565e96257\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws"
Oct 30 17:30:00.433024 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.432943    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b01cc039-2221-4716-8611-8d08004e4778-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n\" (UID: \"b01cc039-2221-4716-8611-8d08004e4778\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n"
Oct 30 17:30:00.433101 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.433081    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5sx2\" (UniqueName: \"kubernetes.io/projected/5668479f-566c-4d0a-9353-7a60f275673a-kube-api-access-n5sx2\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl\" (UID: \"5668479f-566c-4d0a-9353-7a60f275673a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl"
Oct 30 17:30:00.433172 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.433132    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12e09dc3-bfd6-4b8b-90ca-8fab73d0c922-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf\" (UID: \"12e09dc3-bfd6-4b8b-90ca-8fab73d0c922\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf"
Oct 30 17:30:00.433172 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.433161    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5668479f-566c-4d0a-9353-7a60f275673a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl\" (UID: \"5668479f-566c-4d0a-9353-7a60f275673a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl"
Oct 30 17:30:00.433482 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.433458    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqtp9\" (UniqueName: \"kubernetes.io/projected/12e09dc3-bfd6-4b8b-90ca-8fab73d0c922-kube-api-access-gqtp9\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf\" (UID: \"12e09dc3-bfd6-4b8b-90ca-8fab73d0c922\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf"
Oct 30 17:30:00.433598 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.433555    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889690-wzlh5\" (UID: \"786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889690-wzlh5"
Oct 30 17:30:00.433984 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.433946    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl4g9\" (UniqueName: \"kubernetes.io/projected/b01cc039-2221-4716-8611-8d08004e4778-kube-api-access-nl4g9\") pod \"maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n\" (UID: \"b01cc039-2221-4716-8611-8d08004e4778\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n"
Oct 30 17:30:00.434079 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.434003    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt87z\" (UniqueName: \"kubernetes.io/projected/786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3-kube-api-access-gt87z\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889690-wzlh5\" (UID: \"786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889690-wzlh5"
Oct 30 17:30:00.434079 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.434033    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqs75\" (UniqueName: \"kubernetes.io/projected/73952f5d-729f-4433-aeeb-21c31b18c1d9-kube-api-access-dqs75\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99\" (UID: \"73952f5d-729f-4433-aeeb-21c31b18c1d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99"
Oct 30 17:30:00.434207 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.434086    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73952f5d-729f-4433-aeeb-21c31b18c1d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99\" (UID: \"73952f5d-729f-4433-aeeb-21c31b18c1d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99"
Oct 30 17:30:00.434207 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.434124    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbrcz\" (UniqueName: \"kubernetes.io/projected/5d225e1b-96bc-4632-8088-60a96b84999c-kube-api-access-nbrcz\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj\" (UID: \"5d225e1b-96bc-4632-8088-60a96b84999c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj"
Oct 30 17:30:00.434207 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.434162    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d225e1b-96bc-4632-8088-60a96b84999c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj\" (UID: \"5d225e1b-96bc-4632-8088-60a96b84999c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj"
Oct 30 17:30:00.434207 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.434194    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12e09dc3-bfd6-4b8b-90ca-8fab73d0c922-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf\" (UID: \"12e09dc3-bfd6-4b8b-90ca-8fab73d0c922\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf"
Oct 30 17:30:00.435163 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.435140    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d225e1b-96bc-4632-8088-60a96b84999c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj\" (UID: \"5d225e1b-96bc-4632-8088-60a96b84999c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj"
Oct 30 17:30:00.435254 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.435186    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73952f5d-729f-4433-aeeb-21c31b18c1d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99\" (UID: \"73952f5d-729f-4433-aeeb-21c31b18c1d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99"
Oct 30 17:30:00.445114 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.445092    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb"]
Oct 30 17:30:00.445326 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.445308    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m"
Oct 30 17:30:00.453592 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.451777    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v"]
Oct 30 17:30:00.453592 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.453178    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb"
Oct 30 17:30:00.459459 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.459433    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqs75\" (UniqueName: \"kubernetes.io/projected/73952f5d-729f-4433-aeeb-21c31b18c1d9-kube-api-access-dqs75\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99\" (UID: \"73952f5d-729f-4433-aeeb-21c31b18c1d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99"
Oct 30 17:30:00.459667 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.459485    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt87z\" (UniqueName: \"kubernetes.io/projected/786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3-kube-api-access-gt87z\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889690-wzlh5\" (UID: \"786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889690-wzlh5"
Oct 30 17:30:00.462719 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.462697    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbrcz\" (UniqueName: \"kubernetes.io/projected/5d225e1b-96bc-4632-8088-60a96b84999c-kube-api-access-nbrcz\") pod \"maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj\" (UID: \"5d225e1b-96bc-4632-8088-60a96b84999c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj"
Oct 30 17:30:00.474541 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.473739    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqtp9\" (UniqueName: \"kubernetes.io/projected/12e09dc3-bfd6-4b8b-90ca-8fab73d0c922-kube-api-access-gqtp9\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf\" (UID: \"12e09dc3-bfd6-4b8b-90ca-8fab73d0c922\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf"
Oct 30 17:30:00.478363 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.478289    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x989w\" (UniqueName: \"kubernetes.io/projected/c3961d3d-0ada-44af-a530-2a7565e96257-kube-api-access-x989w\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws\" (UID: \"c3961d3d-0ada-44af-a530-2a7565e96257\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws"
Oct 30 17:30:00.482560 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.482533    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m"]
Oct 30 17:30:00.482666 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.482584    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb"]
Oct 30 17:30:00.482666 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.482610    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v"]
Oct 30 17:30:00.482666 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.482634    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889690-5pb4x"]
Oct 30 17:30:00.482827 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.482733    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v"
Oct 30 17:30:00.484425 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.484388    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws"
Oct 30 17:30:00.491811 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.491790    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf"
Oct 30 17:30:00.536891 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.536856    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5sx2\" (UniqueName: \"kubernetes.io/projected/5668479f-566c-4d0a-9353-7a60f275673a-kube-api-access-n5sx2\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl\" (UID: \"5668479f-566c-4d0a-9353-7a60f275673a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl"
Oct 30 17:30:00.537344 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.537316    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5668479f-566c-4d0a-9353-7a60f275673a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl\" (UID: \"5668479f-566c-4d0a-9353-7a60f275673a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl"
Oct 30 17:30:00.538542 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.538517    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5668479f-566c-4d0a-9353-7a60f275673a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl\" (UID: \"5668479f-566c-4d0a-9353-7a60f275673a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl"
Oct 30 17:30:00.538704 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.538683    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl4g9\" (UniqueName: \"kubernetes.io/projected/b01cc039-2221-4716-8611-8d08004e4778-kube-api-access-nl4g9\") pod \"maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n\" (UID: \"b01cc039-2221-4716-8611-8d08004e4778\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n"
Oct 30 17:30:00.538796 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.538779    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w9d9\" (UniqueName: \"kubernetes.io/projected/db5021f4-7307-4efa-8155-dc88fb2d95de-kube-api-access-9w9d9\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m\" (UID: \"db5021f4-7307-4efa-8155-dc88fb2d95de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m"
Oct 30 17:30:00.538843 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.538828    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l45sh\" (UniqueName: \"kubernetes.io/projected/f02fcdb6-55c4-4713-b726-ff1de3d3a7ba-kube-api-access-l45sh\") pod \"maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg\" (UID: \"f02fcdb6-55c4-4713-b726-ff1de3d3a7ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg"
Oct 30 17:30:00.538919 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.538903    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db5021f4-7307-4efa-8155-dc88fb2d95de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m\" (UID: \"db5021f4-7307-4efa-8155-dc88fb2d95de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m"
Oct 30 17:30:00.538956 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.538944    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/993f683e-25cd-4f6e-beff-4da2e7c6ea69-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889690-5pb4x\" (UID: \"993f683e-25cd-4f6e-beff-4da2e7c6ea69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-5pb4x"
Oct 30 17:30:00.539011 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.538996    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9d1683a-8e8a-4717-90ad-a4d84861d2f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v\" (UID: \"c9d1683a-8e8a-4717-90ad-a4d84861d2f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v"
Oct 30 17:30:00.539054 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.539040    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5dzr\" (UniqueName: \"kubernetes.io/projected/35124e4a-306c-4d88-9321-d5de27812912-kube-api-access-z5dzr\") pod \"maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb\" (UID: \"35124e4a-306c-4d88-9321-d5de27812912\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb"
Oct 30 17:30:00.539088 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.539071    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7drh\" (UniqueName: \"kubernetes.io/projected/c9d1683a-8e8a-4717-90ad-a4d84861d2f1-kube-api-access-s7drh\") pod \"maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v\" (UID: \"c9d1683a-8e8a-4717-90ad-a4d84861d2f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v"
Oct 30 17:30:00.539148 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.539135    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35124e4a-306c-4d88-9321-d5de27812912-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb\" (UID: \"35124e4a-306c-4d88-9321-d5de27812912\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb"
Oct 30 17:30:00.539225 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.539211    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cphvf\" (UniqueName: \"kubernetes.io/projected/993f683e-25cd-4f6e-beff-4da2e7c6ea69-kube-api-access-cphvf\") pod \"maas-api-key-cleanup-29889690-5pb4x\" (UID: \"993f683e-25cd-4f6e-beff-4da2e7c6ea69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-5pb4x"
Oct 30 17:30:00.539262 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.539240    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f02fcdb6-55c4-4713-b726-ff1de3d3a7ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg\" (UID: \"f02fcdb6-55c4-4713-b726-ff1de3d3a7ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg"
Oct 30 17:30:00.539295 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.539269    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b01cc039-2221-4716-8611-8d08004e4778-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n\" (UID: \"b01cc039-2221-4716-8611-8d08004e4778\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n"
Oct 30 17:30:00.542448 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.539974    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b01cc039-2221-4716-8611-8d08004e4778-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n\" (UID: \"b01cc039-2221-4716-8611-8d08004e4778\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n"
Oct 30 17:30:00.542448 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.541036    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f02fcdb6-55c4-4713-b726-ff1de3d3a7ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg\" (UID: \"f02fcdb6-55c4-4713-b726-ff1de3d3a7ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg"
Oct 30 17:30:00.542448 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.541191    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/993f683e-25cd-4f6e-beff-4da2e7c6ea69-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889690-5pb4x\" (UID: \"993f683e-25cd-4f6e-beff-4da2e7c6ea69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-5pb4x"
Oct 30 17:30:00.550628 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.549453    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch" podStartSLOduration=2.549436493 podStartE2EDuration="2.549436493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:00.545316377 +0000 UTC m=+2576.980218606" watchObservedRunningTime="2026-10-30 17:30:00.549436493 +0000 UTC m=+2576.984338698"
Oct 30 17:30:00.564890 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.562925    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5sx2\" (UniqueName: \"kubernetes.io/projected/5668479f-566c-4d0a-9353-7a60f275673a-kube-api-access-n5sx2\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl\" (UID: \"5668479f-566c-4d0a-9353-7a60f275673a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl"
Oct 30 17:30:00.569169 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.565737    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cphvf\" (UniqueName: \"kubernetes.io/projected/993f683e-25cd-4f6e-beff-4da2e7c6ea69-kube-api-access-cphvf\") pod \"maas-api-key-cleanup-29889690-5pb4x\" (UID: \"993f683e-25cd-4f6e-beff-4da2e7c6ea69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-5pb4x"
Oct 30 17:30:00.569169 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.568107    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl4g9\" (UniqueName: \"kubernetes.io/projected/b01cc039-2221-4716-8611-8d08004e4778-kube-api-access-nl4g9\") pod \"maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n\" (UID: \"b01cc039-2221-4716-8611-8d08004e4778\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n"
Oct 30 17:30:00.570129 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.570101    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l45sh\" (UniqueName: \"kubernetes.io/projected/f02fcdb6-55c4-4713-b726-ff1de3d3a7ba-kube-api-access-l45sh\") pod \"maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg\" (UID: \"f02fcdb6-55c4-4713-b726-ff1de3d3a7ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg"
Oct 30 17:30:00.640736 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.640698    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9d1683a-8e8a-4717-90ad-a4d84861d2f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v\" (UID: \"c9d1683a-8e8a-4717-90ad-a4d84861d2f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v"
Oct 30 17:30:00.640934 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.640760    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5dzr\" (UniqueName: \"kubernetes.io/projected/35124e4a-306c-4d88-9321-d5de27812912-kube-api-access-z5dzr\") pod \"maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb\" (UID: \"35124e4a-306c-4d88-9321-d5de27812912\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb"
Oct 30 17:30:00.640934 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.640788    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7drh\" (UniqueName: \"kubernetes.io/projected/c9d1683a-8e8a-4717-90ad-a4d84861d2f1-kube-api-access-s7drh\") pod \"maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v\" (UID: \"c9d1683a-8e8a-4717-90ad-a4d84861d2f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v"
Oct 30 17:30:00.640934 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.640848    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35124e4a-306c-4d88-9321-d5de27812912-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb\" (UID: \"35124e4a-306c-4d88-9321-d5de27812912\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb"
Oct 30 17:30:00.641125 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.641097    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w9d9\" (UniqueName: \"kubernetes.io/projected/db5021f4-7307-4efa-8155-dc88fb2d95de-kube-api-access-9w9d9\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m\" (UID: \"db5021f4-7307-4efa-8155-dc88fb2d95de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m"
Oct 30 17:30:00.642226 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.641188    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db5021f4-7307-4efa-8155-dc88fb2d95de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m\" (UID: \"db5021f4-7307-4efa-8155-dc88fb2d95de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m"
Oct 30 17:30:00.642226 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.641579    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9d1683a-8e8a-4717-90ad-a4d84861d2f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v\" (UID: \"c9d1683a-8e8a-4717-90ad-a4d84861d2f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v"
Oct 30 17:30:00.642226 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.642056    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db5021f4-7307-4efa-8155-dc88fb2d95de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m\" (UID: \"db5021f4-7307-4efa-8155-dc88fb2d95de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m"
Oct 30 17:30:00.642226 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.642167    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35124e4a-306c-4d88-9321-d5de27812912-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb\" (UID: \"35124e4a-306c-4d88-9321-d5de27812912\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb"
Oct 30 17:30:00.701509 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.689071    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w9d9\" (UniqueName: \"kubernetes.io/projected/db5021f4-7307-4efa-8155-dc88fb2d95de-kube-api-access-9w9d9\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m\" (UID: \"db5021f4-7307-4efa-8155-dc88fb2d95de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m"
Oct 30 17:30:00.701509 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.690418    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj"
Oct 30 17:30:00.701509 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.697437    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5dzr\" (UniqueName: \"kubernetes.io/projected/35124e4a-306c-4d88-9321-d5de27812912-kube-api-access-z5dzr\") pod \"maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb\" (UID: \"35124e4a-306c-4d88-9321-d5de27812912\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb"
Oct 30 17:30:00.701509 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.697764    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99"
Oct 30 17:30:00.701873 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.701751    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889690-wzlh5"
Oct 30 17:30:00.721433 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.704561    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7drh\" (UniqueName: \"kubernetes.io/projected/c9d1683a-8e8a-4717-90ad-a4d84861d2f1-kube-api-access-s7drh\") pod \"maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v\" (UID: \"c9d1683a-8e8a-4717-90ad-a4d84861d2f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v"
Oct 30 17:30:00.731791 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.731764    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n"
Oct 30 17:30:00.732444 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.732392    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg"
Oct 30 17:30:00.745075 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.745012    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl"
Oct 30 17:30:00.761330 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.761198    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-5pb4x"
Oct 30 17:30:00.809229 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.809195    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws"]
Oct 30 17:30:00.826844 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.826481    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf"]
Oct 30 17:30:00.888118 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.886337    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v"
Oct 30 17:30:00.888118 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.886594    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb"
Oct 30 17:30:00.888118 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:00.886698    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m"
Oct 30 17:30:01.103185 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:01.103153    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889690-wzlh5"]
Oct 30 17:30:01.152706 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:01.152680    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:30:01.162123 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:01.162102    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:30:01.341777 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:01.341742    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889690-wzlh5" event={"ID":"786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3","Type":"ContainerStarted","Data":"218cc0907bd8f2152edb07ab535c938c728fd6cca672a5f7ce825854d86c48e4"}
Oct 30 17:30:01.343046 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:01.343016    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf" event={"ID":"12e09dc3-bfd6-4b8b-90ca-8fab73d0c922","Type":"ContainerStarted","Data":"5160b344f4158ae3ad0f2a0d0ad899c6f6cfeb960c890037c57fa9af18f28aac"}
Oct 30 17:30:01.344461 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:01.344436    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws" event={"ID":"c3961d3d-0ada-44af-a530-2a7565e96257","Type":"ContainerStarted","Data":"a3781df90c03fa2b80971c209460e1ab76a52557dcfce08664a21c4982ea6de5"}
Oct 30 17:30:01.344773 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:01.344754    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:30:01.345721 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:01.345689    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:30:01.371016 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:01.370987    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj"]
Oct 30 17:30:01.371981 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:30:01.371952    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d225e1b_96bc_4632_8088_60a96b84999c.slice/crio-c7388bcbe8510cdd8371c23e4f23e19a98b670027d204473323cb4566c7a0ea5 WatchSource:0}: Error finding container c7388bcbe8510cdd8371c23e4f23e19a98b670027d204473323cb4566c7a0ea5: Status 404 returned error can't find the container with id c7388bcbe8510cdd8371c23e4f23e19a98b670027d204473323cb4566c7a0ea5
Oct 30 17:30:01.384661 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:01.384634    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99"]
Oct 30 17:30:01.386092 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:30:01.386067    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73952f5d_729f_4433_aeeb_21c31b18c1d9.slice/crio-b65ab4af1fd0d9d782b0d8376f7db8e3e1a3c23fd493397f2b690be5cd477d64 WatchSource:0}: Error finding container b65ab4af1fd0d9d782b0d8376f7db8e3e1a3c23fd493397f2b690be5cd477d64: Status 404 returned error can't find the container with id b65ab4af1fd0d9d782b0d8376f7db8e3e1a3c23fd493397f2b690be5cd477d64
Oct 30 17:30:01.401427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:01.397356    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb"]
Oct 30 17:30:01.404709 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:01.404685    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n"]
Oct 30 17:30:01.416197 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:01.416158    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl"]
Oct 30 17:30:01.421224 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:01.421190    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889690-5pb4x"]
Oct 30 17:30:01.448835 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:01.446733    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg"]
Oct 30 17:30:01.452816 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:30:01.452784    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb01cc039_2221_4716_8611_8d08004e4778.slice/crio-22f88cfc3aab643c6560568f348afd071114c8dc2979e028737899dd74713533 WatchSource:0}: Error finding container 22f88cfc3aab643c6560568f348afd071114c8dc2979e028737899dd74713533: Status 404 returned error can't find the container with id 22f88cfc3aab643c6560568f348afd071114c8dc2979e028737899dd74713533
Oct 30 17:30:01.456279 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:30:01.456253    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5668479f_566c_4d0a_9353_7a60f275673a.slice/crio-02b70510d2168c30221c84f53d95bc08d4566f0e37e65612170f08c6e78a5bd8 WatchSource:0}: Error finding container 02b70510d2168c30221c84f53d95bc08d4566f0e37e65612170f08c6e78a5bd8: Status 404 returned error can't find the container with id 02b70510d2168c30221c84f53d95bc08d4566f0e37e65612170f08c6e78a5bd8
Oct 30 17:30:01.494134 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:01.494110    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v"]
Oct 30 17:30:01.537083 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:01.537056    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m"]
Oct 30 17:30:01.553982 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:30:01.553958    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod993f683e_25cd_4f6e_beff_4da2e7c6ea69.slice/crio-8cb938c4f8b808b73688e7f71292e14192bac3dd5e11577c260d9266d139c5ee WatchSource:0}: Error finding container 8cb938c4f8b808b73688e7f71292e14192bac3dd5e11577c260d9266d139c5ee: Status 404 returned error can't find the container with id 8cb938c4f8b808b73688e7f71292e14192bac3dd5e11577c260d9266d139c5ee
Oct 30 17:30:01.656757 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:30:01.656726    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d1683a_8e8a_4717_90ad_a4d84861d2f1.slice/crio-9b2692fb27449882c5481bd627af66bcc63413be87b2714e0c098e01166a05ba WatchSource:0}: Error finding container 9b2692fb27449882c5481bd627af66bcc63413be87b2714e0c098e01166a05ba: Status 404 returned error can't find the container with id 9b2692fb27449882c5481bd627af66bcc63413be87b2714e0c098e01166a05ba
Oct 30 17:30:02.358374 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:02.358335    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n" event={"ID":"b01cc039-2221-4716-8611-8d08004e4778","Type":"ContainerStarted","Data":"940cd52d6dd1b506e29cf0cc2c41a7b92f1f562356ecf6191e9a8a2dbe2ee1ec"}
Oct 30 17:30:02.358821 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:02.358474    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n" event={"ID":"b01cc039-2221-4716-8611-8d08004e4778","Type":"ContainerStarted","Data":"22f88cfc3aab643c6560568f348afd071114c8dc2979e028737899dd74713533"}
Oct 30 17:30:02.360000 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:02.359974    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v" event={"ID":"c9d1683a-8e8a-4717-90ad-a4d84861d2f1","Type":"ContainerStarted","Data":"9b2692fb27449882c5481bd627af66bcc63413be87b2714e0c098e01166a05ba"}
Oct 30 17:30:02.362208 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:02.362179    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb" event={"ID":"35124e4a-306c-4d88-9321-d5de27812912","Type":"ContainerStarted","Data":"3374051763fa9bc9d0cb7bf5b35906e9364a1404a91efeae592d34eb831924bc"}
Oct 30 17:30:02.362315 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:02.362232    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb" event={"ID":"35124e4a-306c-4d88-9321-d5de27812912","Type":"ContainerStarted","Data":"494b7ccc077ce6566ae34871bc177aad1e83b5456f2c32cb8d9ebd8a34437dbb"}
Oct 30 17:30:02.364582 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:02.364127    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99" event={"ID":"73952f5d-729f-4433-aeeb-21c31b18c1d9","Type":"ContainerStarted","Data":"b65ab4af1fd0d9d782b0d8376f7db8e3e1a3c23fd493397f2b690be5cd477d64"}
Oct 30 17:30:02.366072 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:02.366049    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-5pb4x" event={"ID":"993f683e-25cd-4f6e-beff-4da2e7c6ea69","Type":"ContainerStarted","Data":"380853fe52ffa53cac88c77eea003d433fff8f5d3463430c8899d2582fec1f90"}
Oct 30 17:30:02.366172 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:02.366102    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-5pb4x" event={"ID":"993f683e-25cd-4f6e-beff-4da2e7c6ea69","Type":"ContainerStarted","Data":"8cb938c4f8b808b73688e7f71292e14192bac3dd5e11577c260d9266d139c5ee"}
Oct 30 17:30:02.368265 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:02.368241    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj" event={"ID":"5d225e1b-96bc-4632-8088-60a96b84999c","Type":"ContainerStarted","Data":"b7c1fa9a1f9aa891513f0a3cde7e28da643fca1348a591b0c64c755b2802a819"}
Oct 30 17:30:02.368339 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:02.368290    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj" event={"ID":"5d225e1b-96bc-4632-8088-60a96b84999c","Type":"ContainerStarted","Data":"c7388bcbe8510cdd8371c23e4f23e19a98b670027d204473323cb4566c7a0ea5"}
Oct 30 17:30:02.370011 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:02.369988    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg" event={"ID":"f02fcdb6-55c4-4713-b726-ff1de3d3a7ba","Type":"ContainerStarted","Data":"f1e7916b8eae277154984f5cf51cb925902afbabe1401803b325fa431ce41e3a"}
Oct 30 17:30:02.370160 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:02.370141    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg" event={"ID":"f02fcdb6-55c4-4713-b726-ff1de3d3a7ba","Type":"ContainerStarted","Data":"8914be209518b8e5500dd101839d42a62f1768a90d29018ff70d7b0bdfed1549"}
Oct 30 17:30:02.371862 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:02.371837    4433 generic.go:356] "Generic (PLEG): container finished" podID="786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3" containerID="76b6f081a215fc799a3ccfca5b1fd63250b468eefafe5db3cfa02edb4f070fbf" exitCode=60
Oct 30 17:30:02.371951 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:02.371901    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889690-wzlh5" event={"ID":"786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3","Type":"ContainerDied","Data":"76b6f081a215fc799a3ccfca5b1fd63250b468eefafe5db3cfa02edb4f070fbf"}
Oct 30 17:30:02.372105 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:02.372091    4433 scope.go:122] "RemoveContainer" containerID="76b6f081a215fc799a3ccfca5b1fd63250b468eefafe5db3cfa02edb4f070fbf"
Oct 30 17:30:02.373604 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:02.373584    4433 generic.go:356] "Generic (PLEG): container finished" podID="12e09dc3-bfd6-4b8b-90ca-8fab73d0c922" containerID="25a40371ba321cbea34a50157bce2cbf8e62782173e3647f978329cd47f656c5" exitCode=60
Oct 30 17:30:02.373671 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:02.373644    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf" event={"ID":"12e09dc3-bfd6-4b8b-90ca-8fab73d0c922","Type":"ContainerDied","Data":"25a40371ba321cbea34a50157bce2cbf8e62782173e3647f978329cd47f656c5"}
Oct 30 17:30:02.373946 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:02.373923    4433 scope.go:122] "RemoveContainer" containerID="25a40371ba321cbea34a50157bce2cbf8e62782173e3647f978329cd47f656c5"
Oct 30 17:30:02.375849 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:02.375832    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m" event={"ID":"db5021f4-7307-4efa-8155-dc88fb2d95de","Type":"ContainerStarted","Data":"89751aa2d16bfefc4b0a6e883f78e37190bf840007d1ef5b35034370ddc17687"}
Oct 30 17:30:02.375928 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:02.375880    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m" event={"ID":"db5021f4-7307-4efa-8155-dc88fb2d95de","Type":"ContainerStarted","Data":"c3d163290bb01701a8b80ffb240c002089e09b43171378a906a9459dc9e1d1ff"}
Oct 30 17:30:02.377437 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:02.377416    4433 generic.go:356] "Generic (PLEG): container finished" podID="c3961d3d-0ada-44af-a530-2a7565e96257" containerID="59f845f43db29fa21a05467723dd1ae425a5336e6f0f1c1d6fa8a1b5305cd0d6" exitCode=60
Oct 30 17:30:02.377501 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:02.377487    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws" event={"ID":"c3961d3d-0ada-44af-a530-2a7565e96257","Type":"ContainerDied","Data":"59f845f43db29fa21a05467723dd1ae425a5336e6f0f1c1d6fa8a1b5305cd0d6"}
Oct 30 17:30:02.377693 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:02.377677    4433 scope.go:122] "RemoveContainer" containerID="59f845f43db29fa21a05467723dd1ae425a5336e6f0f1c1d6fa8a1b5305cd0d6"
Oct 30 17:30:02.378947 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:02.378926    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl" event={"ID":"5668479f-566c-4d0a-9353-7a60f275673a","Type":"ContainerStarted","Data":"02b70510d2168c30221c84f53d95bc08d4566f0e37e65612170f08c6e78a5bd8"}
Oct 30 17:30:02.400331 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:02.400272    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n" podStartSLOduration=2.400252346 podStartE2EDuration="2.400252346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:02.399010785 +0000 UTC m=+2578.833912990" watchObservedRunningTime="2026-10-30 17:30:02.400252346 +0000 UTC m=+2578.835154551"
Oct 30 17:30:02.442639 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:02.442592    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb" podStartSLOduration=2.442579465 podStartE2EDuration="2.442579465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:02.438727546 +0000 UTC m=+2578.873629751" watchObservedRunningTime="2026-10-30 17:30:02.442579465 +0000 UTC m=+2578.877481718"
Oct 30 17:30:02.481302 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:02.481251    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m" podStartSLOduration=2.481235499 podStartE2EDuration="2.481235499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:02.477326768 +0000 UTC m=+2578.912228972" watchObservedRunningTime="2026-10-30 17:30:02.481235499 +0000 UTC m=+2578.916137703"
Oct 30 17:30:02.520803 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:02.520756    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-5pb4x" podStartSLOduration=2.520743617 podStartE2EDuration="2.520743617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:02.519269337 +0000 UTC m=+2578.954171566" watchObservedRunningTime="2026-10-30 17:30:02.520743617 +0000 UTC m=+2578.955645822"
Oct 30 17:30:02.643974 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:02.643921    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg" podStartSLOduration=2.643906468 podStartE2EDuration="2.643906468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:02.640313887 +0000 UTC m=+2579.075216095" watchObservedRunningTime="2026-10-30 17:30:02.643906468 +0000 UTC m=+2579.078808676"
Oct 30 17:30:02.685753 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:02.685655    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj" podStartSLOduration=2.685641409 podStartE2EDuration="2.685641409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:02.682433752 +0000 UTC m=+2579.117335951" watchObservedRunningTime="2026-10-30 17:30:02.685641409 +0000 UTC m=+2579.120543613"
Oct 30 17:30:03.388539 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.388505    4433 generic.go:356] "Generic (PLEG): container finished" podID="c9d1683a-8e8a-4717-90ad-a4d84861d2f1" containerID="1879594af2666f7b7f5019e32dc240e4aaea9738bc663b613a84de653b6c85bc" exitCode=60
Oct 30 17:30:03.388938 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.388601    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v" event={"ID":"c9d1683a-8e8a-4717-90ad-a4d84861d2f1","Type":"ContainerDied","Data":"1879594af2666f7b7f5019e32dc240e4aaea9738bc663b613a84de653b6c85bc"}
Oct 30 17:30:03.388938 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.388921    4433 scope.go:122] "RemoveContainer" containerID="1879594af2666f7b7f5019e32dc240e4aaea9738bc663b613a84de653b6c85bc"
Oct 30 17:30:03.390375 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.390353    4433 generic.go:356] "Generic (PLEG): container finished" podID="35124e4a-306c-4d88-9321-d5de27812912" containerID="3374051763fa9bc9d0cb7bf5b35906e9364a1404a91efeae592d34eb831924bc" exitCode=60
Oct 30 17:30:03.390517 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.390438    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb" event={"ID":"35124e4a-306c-4d88-9321-d5de27812912","Type":"ContainerDied","Data":"3374051763fa9bc9d0cb7bf5b35906e9364a1404a91efeae592d34eb831924bc"}
Oct 30 17:30:03.390728 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.390707    4433 scope.go:122] "RemoveContainer" containerID="3374051763fa9bc9d0cb7bf5b35906e9364a1404a91efeae592d34eb831924bc"
Oct 30 17:30:03.393499 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.393472    4433 generic.go:356] "Generic (PLEG): container finished" podID="73952f5d-729f-4433-aeeb-21c31b18c1d9" containerID="0db59d62a9b31afacc2d7bf10fcd0f8440df0a7b1b466866890fbc81aa8a6b0a" exitCode=60
Oct 30 17:30:03.393563 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.393532    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99" event={"ID":"73952f5d-729f-4433-aeeb-21c31b18c1d9","Type":"ContainerDied","Data":"0db59d62a9b31afacc2d7bf10fcd0f8440df0a7b1b466866890fbc81aa8a6b0a"}
Oct 30 17:30:03.393962 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.393870    4433 scope.go:122] "RemoveContainer" containerID="0db59d62a9b31afacc2d7bf10fcd0f8440df0a7b1b466866890fbc81aa8a6b0a"
Oct 30 17:30:03.395235 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.395212    4433 generic.go:356] "Generic (PLEG): container finished" podID="993f683e-25cd-4f6e-beff-4da2e7c6ea69" containerID="380853fe52ffa53cac88c77eea003d433fff8f5d3463430c8899d2582fec1f90" exitCode=60
Oct 30 17:30:03.395346 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.395294    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-5pb4x" event={"ID":"993f683e-25cd-4f6e-beff-4da2e7c6ea69","Type":"ContainerDied","Data":"380853fe52ffa53cac88c77eea003d433fff8f5d3463430c8899d2582fec1f90"}
Oct 30 17:30:03.395503 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.395488    4433 scope.go:122] "RemoveContainer" containerID="380853fe52ffa53cac88c77eea003d433fff8f5d3463430c8899d2582fec1f90"
Oct 30 17:30:03.397128 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.397110    4433 generic.go:356] "Generic (PLEG): container finished" podID="5d225e1b-96bc-4632-8088-60a96b84999c" containerID="b7c1fa9a1f9aa891513f0a3cde7e28da643fca1348a591b0c64c755b2802a819" exitCode=60
Oct 30 17:30:03.397194 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.397131    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj" event={"ID":"5d225e1b-96bc-4632-8088-60a96b84999c","Type":"ContainerDied","Data":"b7c1fa9a1f9aa891513f0a3cde7e28da643fca1348a591b0c64c755b2802a819"}
Oct 30 17:30:03.397378 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.397364    4433 scope.go:122] "RemoveContainer" containerID="b7c1fa9a1f9aa891513f0a3cde7e28da643fca1348a591b0c64c755b2802a819"
Oct 30 17:30:03.398868 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.398847    4433 generic.go:356] "Generic (PLEG): container finished" podID="f02fcdb6-55c4-4713-b726-ff1de3d3a7ba" containerID="f1e7916b8eae277154984f5cf51cb925902afbabe1401803b325fa431ce41e3a" exitCode=60
Oct 30 17:30:03.398958 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.398925    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg" event={"ID":"f02fcdb6-55c4-4713-b726-ff1de3d3a7ba","Type":"ContainerDied","Data":"f1e7916b8eae277154984f5cf51cb925902afbabe1401803b325fa431ce41e3a"}
Oct 30 17:30:03.399102 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.399089    4433 scope.go:122] "RemoveContainer" containerID="f1e7916b8eae277154984f5cf51cb925902afbabe1401803b325fa431ce41e3a"
Oct 30 17:30:03.401389 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.401371    4433 generic.go:356] "Generic (PLEG): container finished" podID="786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3" containerID="0b7a652e52157980502a0539ea2a9b3c2d56c784ac8dcf17bc7c6f523a1f809e" exitCode=60
Oct 30 17:30:03.401471 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.401432    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889690-wzlh5" event={"ID":"786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3","Type":"ContainerDied","Data":"0b7a652e52157980502a0539ea2a9b3c2d56c784ac8dcf17bc7c6f523a1f809e"}
Oct 30 17:30:03.401517 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.401480    4433 scope.go:122] "RemoveContainer" containerID="76b6f081a215fc799a3ccfca5b1fd63250b468eefafe5db3cfa02edb4f070fbf"
Oct 30 17:30:03.401741 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.401721    4433 scope.go:122] "RemoveContainer" containerID="0b7a652e52157980502a0539ea2a9b3c2d56c784ac8dcf17bc7c6f523a1f809e"
Oct 30 17:30:03.401998 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:03.401973    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889690-wzlh5_odh-ai-gateway-infra(786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889690-wzlh5" podUID="786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3"
Oct 30 17:30:03.404486 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.404464    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf" event={"ID":"12e09dc3-bfd6-4b8b-90ca-8fab73d0c922","Type":"ContainerStarted","Data":"f4f38d3569f383579f89c799ac636b53e8445561645dd675fb5acd275e5bf48b"}
Oct 30 17:30:03.406469 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.406442    4433 generic.go:356] "Generic (PLEG): container finished" podID="db5021f4-7307-4efa-8155-dc88fb2d95de" containerID="89751aa2d16bfefc4b0a6e883f78e37190bf840007d1ef5b35034370ddc17687" exitCode=60
Oct 30 17:30:03.406591 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.406572    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m" event={"ID":"db5021f4-7307-4efa-8155-dc88fb2d95de","Type":"ContainerDied","Data":"89751aa2d16bfefc4b0a6e883f78e37190bf840007d1ef5b35034370ddc17687"}
Oct 30 17:30:03.406838 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.406816    4433 scope.go:122] "RemoveContainer" containerID="89751aa2d16bfefc4b0a6e883f78e37190bf840007d1ef5b35034370ddc17687"
Oct 30 17:30:03.409760 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.409735    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws" event={"ID":"c3961d3d-0ada-44af-a530-2a7565e96257","Type":"ContainerStarted","Data":"41cfb757fbb61c0d546efc6771ecce3f86b38a27cb2c4ee4903ca8a5c9709dec"}
Oct 30 17:30:03.412834 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.412697    4433 generic.go:356] "Generic (PLEG): container finished" podID="5668479f-566c-4d0a-9353-7a60f275673a" containerID="1dad14b50ebc2996dd5d3aa39647b2ec9dac9597ea51efbd9df1ae9fafe55a9e" exitCode=60
Oct 30 17:30:03.412920 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.412841    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl" event={"ID":"5668479f-566c-4d0a-9353-7a60f275673a","Type":"ContainerDied","Data":"1dad14b50ebc2996dd5d3aa39647b2ec9dac9597ea51efbd9df1ae9fafe55a9e"}
Oct 30 17:30:03.413195 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.413119    4433 scope.go:122] "RemoveContainer" containerID="1dad14b50ebc2996dd5d3aa39647b2ec9dac9597ea51efbd9df1ae9fafe55a9e"
Oct 30 17:30:03.414566 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.414544    4433 generic.go:356] "Generic (PLEG): container finished" podID="b01cc039-2221-4716-8611-8d08004e4778" containerID="940cd52d6dd1b506e29cf0cc2c41a7b92f1f562356ecf6191e9a8a2dbe2ee1ec" exitCode=60
Oct 30 17:30:03.415820 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.415500    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n" event={"ID":"b01cc039-2221-4716-8611-8d08004e4778","Type":"ContainerDied","Data":"940cd52d6dd1b506e29cf0cc2c41a7b92f1f562356ecf6191e9a8a2dbe2ee1ec"}
Oct 30 17:30:03.415820 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.415514    4433 scope.go:122] "RemoveContainer" containerID="940cd52d6dd1b506e29cf0cc2c41a7b92f1f562356ecf6191e9a8a2dbe2ee1ec"
Oct 30 17:30:03.431984 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.431942    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl" podStartSLOduration=3.431928282 podStartE2EDuration="3.431928282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:02.76465696 +0000 UTC m=+2579.199559166" watchObservedRunningTime="2026-10-30 17:30:03.431928282 +0000 UTC m=+2579.866830487"
Oct 30 17:30:03.508614 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.508566    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws" podStartSLOduration=3.508550733 podStartE2EDuration="3.508550733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:03.507646807 +0000 UTC m=+2579.942549240" watchObservedRunningTime="2026-10-30 17:30:03.508550733 +0000 UTC m=+2579.943452939"
Oct 30 17:30:03.801798 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:03.801746    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf" podStartSLOduration=3.801731664 podStartE2EDuration="3.801731664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:03.798902938 +0000 UTC m=+2580.233805143" watchObservedRunningTime="2026-10-30 17:30:03.801731664 +0000 UTC m=+2580.236633909"
Oct 30 17:30:04.423378 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:04.423284    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n" event={"ID":"b01cc039-2221-4716-8611-8d08004e4778","Type":"ContainerStarted","Data":"d03dbb888e331a91797ddac825826477a5311d5ac520e3676e10f71731dc79c8"}
Oct 30 17:30:04.425346 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:04.425319    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v" event={"ID":"c9d1683a-8e8a-4717-90ad-a4d84861d2f1","Type":"ContainerStarted","Data":"d6ec3e884af85d74a090b921c48ca562bbd5e47c3de80d2e019f0bbb693e7796"}
Oct 30 17:30:04.427149 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:04.427124    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb" event={"ID":"35124e4a-306c-4d88-9321-d5de27812912","Type":"ContainerStarted","Data":"cb3c1bbf7d378f6322609f3cdc6350e03d9eadd6dac9c6df6216f1ff244a04c9"}
Oct 30 17:30:04.428900 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:04.428878    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99" event={"ID":"73952f5d-729f-4433-aeeb-21c31b18c1d9","Type":"ContainerStarted","Data":"61b24818b271e1f6b00f8e559b31886f0c08b22a46dd82444a65da1a5313c4bf"}
Oct 30 17:30:04.430665 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:04.430643    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-5pb4x" event={"ID":"993f683e-25cd-4f6e-beff-4da2e7c6ea69","Type":"ContainerStarted","Data":"e10a5fc36b8194e37be5063bb19bcb3a6b72199bbb9d15fbbabed9ef9d833090"}
Oct 30 17:30:04.432458 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:04.432434    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj" event={"ID":"5d225e1b-96bc-4632-8088-60a96b84999c","Type":"ContainerStarted","Data":"098fa0c59fc3f8e8336dffedbc72fcec39068df718ef603dc8f17a6a9be1e73a"}
Oct 30 17:30:04.434265 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:04.434244    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg" event={"ID":"f02fcdb6-55c4-4713-b726-ff1de3d3a7ba","Type":"ContainerStarted","Data":"b6e139cfe9fb0aabb2ea36d6ac7e2be78a2d5681fe38e6fd9bd1dcec72f84134"}
Oct 30 17:30:04.435949 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:04.435931    4433 scope.go:122] "RemoveContainer" containerID="0b7a652e52157980502a0539ea2a9b3c2d56c784ac8dcf17bc7c6f523a1f809e"
Oct 30 17:30:04.436183 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:04.436163    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889690-wzlh5_odh-ai-gateway-infra(786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889690-wzlh5" podUID="786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3"
Oct 30 17:30:04.437563 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:04.437543    4433 generic.go:356] "Generic (PLEG): container finished" podID="12e09dc3-bfd6-4b8b-90ca-8fab73d0c922" containerID="f4f38d3569f383579f89c799ac636b53e8445561645dd675fb5acd275e5bf48b" exitCode=60
Oct 30 17:30:04.437646 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:04.437609    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf" event={"ID":"12e09dc3-bfd6-4b8b-90ca-8fab73d0c922","Type":"ContainerDied","Data":"f4f38d3569f383579f89c799ac636b53e8445561645dd675fb5acd275e5bf48b"}
Oct 30 17:30:04.437692 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:04.437648    4433 scope.go:122] "RemoveContainer" containerID="25a40371ba321cbea34a50157bce2cbf8e62782173e3647f978329cd47f656c5"
Oct 30 17:30:04.437812 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:04.437793    4433 scope.go:122] "RemoveContainer" containerID="f4f38d3569f383579f89c799ac636b53e8445561645dd675fb5acd275e5bf48b"
Oct 30 17:30:04.438077 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:04.438045    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf_odh-ai-gateway-infra(12e09dc3-bfd6-4b8b-90ca-8fab73d0c922)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf" podUID="12e09dc3-bfd6-4b8b-90ca-8fab73d0c922"
Oct 30 17:30:04.439488 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:04.439458    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m" event={"ID":"db5021f4-7307-4efa-8155-dc88fb2d95de","Type":"ContainerStarted","Data":"850362c6459442a22f5647c95f834c9cf172347affd4d8ca6745854a5b8e3ad9"}
Oct 30 17:30:04.441164 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:04.441145    4433 generic.go:356] "Generic (PLEG): container finished" podID="c3961d3d-0ada-44af-a530-2a7565e96257" containerID="41cfb757fbb61c0d546efc6771ecce3f86b38a27cb2c4ee4903ca8a5c9709dec" exitCode=60
Oct 30 17:30:04.441267 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:04.441208    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws" event={"ID":"c3961d3d-0ada-44af-a530-2a7565e96257","Type":"ContainerDied","Data":"41cfb757fbb61c0d546efc6771ecce3f86b38a27cb2c4ee4903ca8a5c9709dec"}
Oct 30 17:30:04.441392 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:04.441377    4433 scope.go:122] "RemoveContainer" containerID="41cfb757fbb61c0d546efc6771ecce3f86b38a27cb2c4ee4903ca8a5c9709dec"
Oct 30 17:30:04.441686 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:04.441654    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws_odh-ai-gateway-infra(c3961d3d-0ada-44af-a530-2a7565e96257)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws" podUID="c3961d3d-0ada-44af-a530-2a7565e96257"
Oct 30 17:30:04.443082 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:04.443059    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl" event={"ID":"5668479f-566c-4d0a-9353-7a60f275673a","Type":"ContainerStarted","Data":"e682c9bef44ab1f46c74ce5db0941719bb488ef6dc902060c0ec2be72df48d3e"}
Oct 30 17:30:04.451640 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:04.451620    4433 scope.go:122] "RemoveContainer" containerID="59f845f43db29fa21a05467723dd1ae425a5336e6f0f1c1d6fa8a1b5305cd0d6"
Oct 30 17:30:04.787494 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:04.787324    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v" podStartSLOduration=4.787309682 podStartE2EDuration="4.787309682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:04.786852869 +0000 UTC m=+2581.221755077" watchObservedRunningTime="2026-10-30 17:30:04.787309682 +0000 UTC m=+2581.222211887"
Oct 30 17:30:04.963417 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:04.963351    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99" podStartSLOduration=4.9633358770000005 podStartE2EDuration="4.963335877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:04.959795512 +0000 UTC m=+2581.394697718" watchObservedRunningTime="2026-10-30 17:30:04.963335877 +0000 UTC m=+2581.398238124"
Oct 30 17:30:05.201479 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.198455    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3260f8-f9c947b48-xn892"]
Oct 30 17:30:05.217724 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.217688    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-f9c947b48-xn892"
Oct 30 17:30:05.232747 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.232717    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3260f8\""
Oct 30 17:30:05.233560 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.233022    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3260f8-dockercfg-tt26v\""
Oct 30 17:30:05.234205 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.234186    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3260f8-f9c947b48-xn892"]
Oct 30 17:30:05.303429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.288515    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a8d678-3fd9-4ce9-9e9f-f123f42dda10-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3260f8-f9c947b48-xn892\" (UID: \"d0a8d678-3fd9-4ce9-9e9f-f123f42dda10\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-f9c947b48-xn892"
Oct 30 17:30:05.303429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.300960    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5grmq\" (UniqueName: \"kubernetes.io/projected/d0a8d678-3fd9-4ce9-9e9f-f123f42dda10-kube-api-access-5grmq\") pod \"payload-processing-e2e-ipp-b-3260f8-f9c947b48-xn892\" (UID: \"d0a8d678-3fd9-4ce9-9e9f-f123f42dda10\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-f9c947b48-xn892"
Oct 30 17:30:05.404363 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.404328    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a8d678-3fd9-4ce9-9e9f-f123f42dda10-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3260f8-f9c947b48-xn892\" (UID: \"d0a8d678-3fd9-4ce9-9e9f-f123f42dda10\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-f9c947b48-xn892"
Oct 30 17:30:05.405283 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.405260    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5grmq\" (UniqueName: \"kubernetes.io/projected/d0a8d678-3fd9-4ce9-9e9f-f123f42dda10-kube-api-access-5grmq\") pod \"payload-processing-e2e-ipp-b-3260f8-f9c947b48-xn892\" (UID: \"d0a8d678-3fd9-4ce9-9e9f-f123f42dda10\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-f9c947b48-xn892"
Oct 30 17:30:05.407123 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.407104    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a8d678-3fd9-4ce9-9e9f-f123f42dda10-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3260f8-f9c947b48-xn892\" (UID: \"d0a8d678-3fd9-4ce9-9e9f-f123f42dda10\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-f9c947b48-xn892"
Oct 30 17:30:05.436001 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.435926    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5grmq\" (UniqueName: \"kubernetes.io/projected/d0a8d678-3fd9-4ce9-9e9f-f123f42dda10-kube-api-access-5grmq\") pod \"payload-processing-e2e-ipp-b-3260f8-f9c947b48-xn892\" (UID: \"d0a8d678-3fd9-4ce9-9e9f-f123f42dda10\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-f9c947b48-xn892"
Oct 30 17:30:05.456875 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.456796    4433 generic.go:356] "Generic (PLEG): container finished" podID="b01cc039-2221-4716-8611-8d08004e4778" containerID="d03dbb888e331a91797ddac825826477a5311d5ac520e3676e10f71731dc79c8" exitCode=60
Oct 30 17:30:05.457041 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.456881    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n" event={"ID":"b01cc039-2221-4716-8611-8d08004e4778","Type":"ContainerDied","Data":"d03dbb888e331a91797ddac825826477a5311d5ac520e3676e10f71731dc79c8"}
Oct 30 17:30:05.457041 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.456950    4433 scope.go:122] "RemoveContainer" containerID="940cd52d6dd1b506e29cf0cc2c41a7b92f1f562356ecf6191e9a8a2dbe2ee1ec"
Oct 30 17:30:05.457236 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.457220    4433 scope.go:122] "RemoveContainer" containerID="d03dbb888e331a91797ddac825826477a5311d5ac520e3676e10f71731dc79c8"
Oct 30 17:30:05.457614 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:05.457591    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n_odh-ai-gateway-infra(b01cc039-2221-4716-8611-8d08004e4778)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n" podUID="b01cc039-2221-4716-8611-8d08004e4778"
Oct 30 17:30:05.459119 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.459098    4433 generic.go:356] "Generic (PLEG): container finished" podID="c9d1683a-8e8a-4717-90ad-a4d84861d2f1" containerID="d6ec3e884af85d74a090b921c48ca562bbd5e47c3de80d2e019f0bbb693e7796" exitCode=60
Oct 30 17:30:05.459210 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.459185    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v" event={"ID":"c9d1683a-8e8a-4717-90ad-a4d84861d2f1","Type":"ContainerDied","Data":"d6ec3e884af85d74a090b921c48ca562bbd5e47c3de80d2e019f0bbb693e7796"}
Oct 30 17:30:05.459475 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.459450    4433 scope.go:122] "RemoveContainer" containerID="d6ec3e884af85d74a090b921c48ca562bbd5e47c3de80d2e019f0bbb693e7796"
Oct 30 17:30:05.459822 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:05.459800    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v_odh-ai-gateway-infra(c9d1683a-8e8a-4717-90ad-a4d84861d2f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v" podUID="c9d1683a-8e8a-4717-90ad-a4d84861d2f1"
Oct 30 17:30:05.460995 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.460978    4433 generic.go:356] "Generic (PLEG): container finished" podID="35124e4a-306c-4d88-9321-d5de27812912" containerID="cb3c1bbf7d378f6322609f3cdc6350e03d9eadd6dac9c6df6216f1ff244a04c9" exitCode=60
Oct 30 17:30:05.461093 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.461025    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb" event={"ID":"35124e4a-306c-4d88-9321-d5de27812912","Type":"ContainerDied","Data":"cb3c1bbf7d378f6322609f3cdc6350e03d9eadd6dac9c6df6216f1ff244a04c9"}
Oct 30 17:30:05.461320 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.461300    4433 scope.go:122] "RemoveContainer" containerID="cb3c1bbf7d378f6322609f3cdc6350e03d9eadd6dac9c6df6216f1ff244a04c9"
Oct 30 17:30:05.461699 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:05.461667    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb_odh-ai-gateway-infra(35124e4a-306c-4d88-9321-d5de27812912)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb" podUID="35124e4a-306c-4d88-9321-d5de27812912"
Oct 30 17:30:05.462943 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.462921    4433 generic.go:356] "Generic (PLEG): container finished" podID="73952f5d-729f-4433-aeeb-21c31b18c1d9" containerID="61b24818b271e1f6b00f8e559b31886f0c08b22a46dd82444a65da1a5313c4bf" exitCode=60
Oct 30 17:30:05.463040 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.463003    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99" event={"ID":"73952f5d-729f-4433-aeeb-21c31b18c1d9","Type":"ContainerDied","Data":"61b24818b271e1f6b00f8e559b31886f0c08b22a46dd82444a65da1a5313c4bf"}
Oct 30 17:30:05.463232 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.463216    4433 scope.go:122] "RemoveContainer" containerID="61b24818b271e1f6b00f8e559b31886f0c08b22a46dd82444a65da1a5313c4bf"
Oct 30 17:30:05.463542 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:05.463525    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99_odh-ai-gateway-infra(73952f5d-729f-4433-aeeb-21c31b18c1d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99" podUID="73952f5d-729f-4433-aeeb-21c31b18c1d9"
Oct 30 17:30:05.464988 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.464966    4433 generic.go:356] "Generic (PLEG): container finished" podID="993f683e-25cd-4f6e-beff-4da2e7c6ea69" containerID="e10a5fc36b8194e37be5063bb19bcb3a6b72199bbb9d15fbbabed9ef9d833090" exitCode=60
Oct 30 17:30:05.465069 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.464993    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-5pb4x" event={"ID":"993f683e-25cd-4f6e-beff-4da2e7c6ea69","Type":"ContainerDied","Data":"e10a5fc36b8194e37be5063bb19bcb3a6b72199bbb9d15fbbabed9ef9d833090"}
Oct 30 17:30:05.465311 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.465292    4433 scope.go:122] "RemoveContainer" containerID="e10a5fc36b8194e37be5063bb19bcb3a6b72199bbb9d15fbbabed9ef9d833090"
Oct 30 17:30:05.465642 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:05.465616    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889690-5pb4x_odh-ai-gateway-infra(993f683e-25cd-4f6e-beff-4da2e7c6ea69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-5pb4x" podUID="993f683e-25cd-4f6e-beff-4da2e7c6ea69"
Oct 30 17:30:05.467043 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.467022    4433 generic.go:356] "Generic (PLEG): container finished" podID="5d225e1b-96bc-4632-8088-60a96b84999c" containerID="098fa0c59fc3f8e8336dffedbc72fcec39068df718ef603dc8f17a6a9be1e73a" exitCode=60
Oct 30 17:30:05.467154 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.467109    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj" event={"ID":"5d225e1b-96bc-4632-8088-60a96b84999c","Type":"ContainerDied","Data":"098fa0c59fc3f8e8336dffedbc72fcec39068df718ef603dc8f17a6a9be1e73a"}
Oct 30 17:30:05.467309 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.467295    4433 scope.go:122] "RemoveContainer" containerID="098fa0c59fc3f8e8336dffedbc72fcec39068df718ef603dc8f17a6a9be1e73a"
Oct 30 17:30:05.467614 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:05.467596    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj_odh-ai-gateway-infra(5d225e1b-96bc-4632-8088-60a96b84999c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj" podUID="5d225e1b-96bc-4632-8088-60a96b84999c"
Oct 30 17:30:05.469145 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.469120    4433 generic.go:356] "Generic (PLEG): container finished" podID="f02fcdb6-55c4-4713-b726-ff1de3d3a7ba" containerID="b6e139cfe9fb0aabb2ea36d6ac7e2be78a2d5681fe38e6fd9bd1dcec72f84134" exitCode=60
Oct 30 17:30:05.469234 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.469219    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg" event={"ID":"f02fcdb6-55c4-4713-b726-ff1de3d3a7ba","Type":"ContainerDied","Data":"b6e139cfe9fb0aabb2ea36d6ac7e2be78a2d5681fe38e6fd9bd1dcec72f84134"}
Oct 30 17:30:05.469418 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.469390    4433 scope.go:122] "RemoveContainer" containerID="b6e139cfe9fb0aabb2ea36d6ac7e2be78a2d5681fe38e6fd9bd1dcec72f84134"
Oct 30 17:30:05.469791 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:05.469768    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg_odh-ai-gateway-infra(f02fcdb6-55c4-4713-b726-ff1de3d3a7ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg" podUID="f02fcdb6-55c4-4713-b726-ff1de3d3a7ba"
Oct 30 17:30:05.471151 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.471133    4433 scope.go:122] "RemoveContainer" containerID="f4f38d3569f383579f89c799ac636b53e8445561645dd675fb5acd275e5bf48b"
Oct 30 17:30:05.471399 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:05.471382    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf_odh-ai-gateway-infra(12e09dc3-bfd6-4b8b-90ca-8fab73d0c922)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf" podUID="12e09dc3-bfd6-4b8b-90ca-8fab73d0c922"
Oct 30 17:30:05.472093 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.472077    4433 scope.go:122] "RemoveContainer" containerID="1879594af2666f7b7f5019e32dc240e4aaea9738bc663b613a84de653b6c85bc"
Oct 30 17:30:05.472781 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.472759    4433 generic.go:356] "Generic (PLEG): container finished" podID="db5021f4-7307-4efa-8155-dc88fb2d95de" containerID="850362c6459442a22f5647c95f834c9cf172347affd4d8ca6745854a5b8e3ad9" exitCode=60
Oct 30 17:30:05.472873 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.472835    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m" event={"ID":"db5021f4-7307-4efa-8155-dc88fb2d95de","Type":"ContainerDied","Data":"850362c6459442a22f5647c95f834c9cf172347affd4d8ca6745854a5b8e3ad9"}
Oct 30 17:30:05.473038 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.473021    4433 scope.go:122] "RemoveContainer" containerID="850362c6459442a22f5647c95f834c9cf172347affd4d8ca6745854a5b8e3ad9"
Oct 30 17:30:05.473326 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:05.473308    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m_odh-ai-gateway-infra(db5021f4-7307-4efa-8155-dc88fb2d95de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m" podUID="db5021f4-7307-4efa-8155-dc88fb2d95de"
Oct 30 17:30:05.474647 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.474629    4433 scope.go:122] "RemoveContainer" containerID="41cfb757fbb61c0d546efc6771ecce3f86b38a27cb2c4ee4903ca8a5c9709dec"
Oct 30 17:30:05.474863 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:05.474843    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws_odh-ai-gateway-infra(c3961d3d-0ada-44af-a530-2a7565e96257)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws" podUID="c3961d3d-0ada-44af-a530-2a7565e96257"
Oct 30 17:30:05.476113 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.476095    4433 generic.go:356] "Generic (PLEG): container finished" podID="5668479f-566c-4d0a-9353-7a60f275673a" containerID="e682c9bef44ab1f46c74ce5db0941719bb488ef6dc902060c0ec2be72df48d3e" exitCode=60
Oct 30 17:30:05.476212 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.476143    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl" event={"ID":"5668479f-566c-4d0a-9353-7a60f275673a","Type":"ContainerDied","Data":"e682c9bef44ab1f46c74ce5db0941719bb488ef6dc902060c0ec2be72df48d3e"}
Oct 30 17:30:05.476341 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.476328    4433 scope.go:122] "RemoveContainer" containerID="e682c9bef44ab1f46c74ce5db0941719bb488ef6dc902060c0ec2be72df48d3e"
Oct 30 17:30:05.476542 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:05.476527    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl_odh-ai-gateway-infra(5668479f-566c-4d0a-9353-7a60f275673a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl" podUID="5668479f-566c-4d0a-9353-7a60f275673a"
Oct 30 17:30:05.485126 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.485107    4433 scope.go:122] "RemoveContainer" containerID="3374051763fa9bc9d0cb7bf5b35906e9364a1404a91efeae592d34eb831924bc"
Oct 30 17:30:05.498261 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.498236    4433 scope.go:122] "RemoveContainer" containerID="0db59d62a9b31afacc2d7bf10fcd0f8440df0a7b1b466866890fbc81aa8a6b0a"
Oct 30 17:30:05.511967 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.511949    4433 scope.go:122] "RemoveContainer" containerID="380853fe52ffa53cac88c77eea003d433fff8f5d3463430c8899d2582fec1f90"
Oct 30 17:30:05.524265 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.524242    4433 scope.go:122] "RemoveContainer" containerID="b7c1fa9a1f9aa891513f0a3cde7e28da643fca1348a591b0c64c755b2802a819"
Oct 30 17:30:05.539093 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.539071    4433 scope.go:122] "RemoveContainer" containerID="f1e7916b8eae277154984f5cf51cb925902afbabe1401803b325fa431ce41e3a"
Oct 30 17:30:05.553011 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.552993    4433 scope.go:122] "RemoveContainer" containerID="89751aa2d16bfefc4b0a6e883f78e37190bf840007d1ef5b35034370ddc17687"
Oct 30 17:30:05.565880 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.565859    4433 scope.go:122] "RemoveContainer" containerID="1dad14b50ebc2996dd5d3aa39647b2ec9dac9597ea51efbd9df1ae9fafe55a9e"
Oct 30 17:30:05.726493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:05.726412    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-f9c947b48-xn892"
Oct 30 17:30:06.045691 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:06.045663    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3260f8-f9c947b48-xn892"]
Oct 30 17:30:06.047571 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:30:06.047539    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a8d678_3fd9_4ce9_9e9f_f123f42dda10.slice/crio-9b5d1f319bd1f3b168349a038755e43e4411518eb1581e2a29b69025fefd1dc5 WatchSource:0}: Error finding container 9b5d1f319bd1f3b168349a038755e43e4411518eb1581e2a29b69025fefd1dc5: Status 404 returned error can't find the container with id 9b5d1f319bd1f3b168349a038755e43e4411518eb1581e2a29b69025fefd1dc5
Oct 30 17:30:06.484040 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:06.484006    4433 scope.go:122] "RemoveContainer" containerID="d6ec3e884af85d74a090b921c48ca562bbd5e47c3de80d2e019f0bbb693e7796"
Oct 30 17:30:06.484498 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:06.484395    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v_odh-ai-gateway-infra(c9d1683a-8e8a-4717-90ad-a4d84861d2f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v" podUID="c9d1683a-8e8a-4717-90ad-a4d84861d2f1"
Oct 30 17:30:06.485785 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:06.485755    4433 scope.go:122] "RemoveContainer" containerID="cb3c1bbf7d378f6322609f3cdc6350e03d9eadd6dac9c6df6216f1ff244a04c9"
Oct 30 17:30:06.486102 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:06.486082    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb_odh-ai-gateway-infra(35124e4a-306c-4d88-9321-d5de27812912)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb" podUID="35124e4a-306c-4d88-9321-d5de27812912"
Oct 30 17:30:06.487208 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:06.487177    4433 scope.go:122] "RemoveContainer" containerID="61b24818b271e1f6b00f8e559b31886f0c08b22a46dd82444a65da1a5313c4bf"
Oct 30 17:30:06.487436 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:06.487416    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99_odh-ai-gateway-infra(73952f5d-729f-4433-aeeb-21c31b18c1d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99" podUID="73952f5d-729f-4433-aeeb-21c31b18c1d9"
Oct 30 17:30:06.488723 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:06.488696    4433 scope.go:122] "RemoveContainer" containerID="e10a5fc36b8194e37be5063bb19bcb3a6b72199bbb9d15fbbabed9ef9d833090"
Oct 30 17:30:06.488936 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:06.488918    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889690-5pb4x_odh-ai-gateway-infra(993f683e-25cd-4f6e-beff-4da2e7c6ea69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-5pb4x" podUID="993f683e-25cd-4f6e-beff-4da2e7c6ea69"
Oct 30 17:30:06.490127 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:06.490108    4433 scope.go:122] "RemoveContainer" containerID="098fa0c59fc3f8e8336dffedbc72fcec39068df718ef603dc8f17a6a9be1e73a"
Oct 30 17:30:06.490338 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:06.490316    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj_odh-ai-gateway-infra(5d225e1b-96bc-4632-8088-60a96b84999c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj" podUID="5d225e1b-96bc-4632-8088-60a96b84999c"
Oct 30 17:30:06.491645 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:06.491623    4433 scope.go:122] "RemoveContainer" containerID="b6e139cfe9fb0aabb2ea36d6ac7e2be78a2d5681fe38e6fd9bd1dcec72f84134"
Oct 30 17:30:06.491911 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:06.491886    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg_odh-ai-gateway-infra(f02fcdb6-55c4-4713-b726-ff1de3d3a7ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg" podUID="f02fcdb6-55c4-4713-b726-ff1de3d3a7ba"
Oct 30 17:30:06.493239 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:06.493221    4433 scope.go:122] "RemoveContainer" containerID="850362c6459442a22f5647c95f834c9cf172347affd4d8ca6745854a5b8e3ad9"
Oct 30 17:30:06.493509 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:06.493480    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m_odh-ai-gateway-infra(db5021f4-7307-4efa-8155-dc88fb2d95de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m" podUID="db5021f4-7307-4efa-8155-dc88fb2d95de"
Oct 30 17:30:06.494898 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:06.494882    4433 scope.go:122] "RemoveContainer" containerID="e682c9bef44ab1f46c74ce5db0941719bb488ef6dc902060c0ec2be72df48d3e"
Oct 30 17:30:06.495107 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:06.495088    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl_odh-ai-gateway-infra(5668479f-566c-4d0a-9353-7a60f275673a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl" podUID="5668479f-566c-4d0a-9353-7a60f275673a"
Oct 30 17:30:06.496442 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:06.496421    4433 scope.go:122] "RemoveContainer" containerID="d03dbb888e331a91797ddac825826477a5311d5ac520e3676e10f71731dc79c8"
Oct 30 17:30:06.496643 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:06.496625    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n_odh-ai-gateway-infra(b01cc039-2221-4716-8611-8d08004e4778)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n" podUID="b01cc039-2221-4716-8611-8d08004e4778"
Oct 30 17:30:06.497686 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:06.497663    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-f9c947b48-xn892" event={"ID":"d0a8d678-3fd9-4ce9-9e9f-f123f42dda10","Type":"ContainerStarted","Data":"104e854abe6ebc1f9025271dbd5b9f771bbbde6aa25f4cd031879a71bb0cb5cd"}
Oct 30 17:30:06.497777 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:06.497735    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-f9c947b48-xn892" event={"ID":"d0a8d678-3fd9-4ce9-9e9f-f123f42dda10","Type":"ContainerStarted","Data":"9b5d1f319bd1f3b168349a038755e43e4411518eb1581e2a29b69025fefd1dc5"}
Oct 30 17:30:06.497777 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:06.497757    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-f9c947b48-xn892"
Oct 30 17:30:06.887817 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:06.887702    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-f9c947b48-xn892" podStartSLOduration=1.887687284 podStartE2EDuration="1.887687284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:06.886208819 +0000 UTC m=+2583.321111016" watchObservedRunningTime="2026-10-30 17:30:06.887687284 +0000 UTC m=+2583.322589489"
Oct 30 17:30:15.695254 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:15.695224    4433 scope.go:122] "RemoveContainer" containerID="850362c6459442a22f5647c95f834c9cf172347affd4d8ca6745854a5b8e3ad9"
Oct 30 17:30:15.695855 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:15.695826    4433 scope.go:122] "RemoveContainer" containerID="d6ec3e884af85d74a090b921c48ca562bbd5e47c3de80d2e019f0bbb693e7796"
Oct 30 17:30:15.696393 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:15.696365    4433 scope.go:122] "RemoveContainer" containerID="0b7a652e52157980502a0539ea2a9b3c2d56c784ac8dcf17bc7c6f523a1f809e"
Oct 30 17:30:15.697765 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:15.696841    4433 scope.go:122] "RemoveContainer" containerID="f4f38d3569f383579f89c799ac636b53e8445561645dd675fb5acd275e5bf48b"
Oct 30 17:30:15.697765 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:15.697159    4433 scope.go:122] "RemoveContainer" containerID="e10a5fc36b8194e37be5063bb19bcb3a6b72199bbb9d15fbbabed9ef9d833090"
Oct 30 17:30:16.569584 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:16.569507    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v" event={"ID":"c9d1683a-8e8a-4717-90ad-a4d84861d2f1","Type":"ContainerStarted","Data":"379a1235904a64c1c63a7fcddf9ac91e3345b431791f743016d0ca5eee7353f8"}
Oct 30 17:30:16.571461 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:16.571432    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-5pb4x" event={"ID":"993f683e-25cd-4f6e-beff-4da2e7c6ea69","Type":"ContainerStarted","Data":"c13587b489fd232e5de77f5e81f794d7789ce38f76b0e23be357776c34c08201"}
Oct 30 17:30:16.573246 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:16.573220    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889690-wzlh5" event={"ID":"786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3","Type":"ContainerStarted","Data":"7e8c2d2f28179df3ebd544949888e4fc9070edccb5473ce59c2a8941b0af8625"}
Oct 30 17:30:16.575234 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:16.575208    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf" event={"ID":"12e09dc3-bfd6-4b8b-90ca-8fab73d0c922","Type":"ContainerStarted","Data":"b1a18f7cb731a67f023afd08a39ca7d4f5cd681fbab087566d78ad47251079a7"}
Oct 30 17:30:16.577000 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:16.576981    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m" event={"ID":"db5021f4-7307-4efa-8155-dc88fb2d95de","Type":"ContainerStarted","Data":"8029cea349b7e154700b07f83da20617d7bb3e3c2767ac42d923e2e914799bc3"}
Oct 30 17:30:16.612166 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:16.612137    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq"]
Oct 30 17:30:16.617878 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:16.617849    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq"
Oct 30 17:30:16.644243 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:16.637055    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-b87f3c\"/\"default-dockercfg-sq78m\""
Oct 30 17:30:16.644243 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:16.637219    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-b87f3c\"/\"kube-root-ca.crt\""
Oct 30 17:30:16.644243 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:16.637304    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-b87f3c\"/\"openshift-service-ca.crt\""
Oct 30 17:30:16.644243 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:16.638305    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-b87f3c\"/\"auth-test-model-b87f3c-kserve-self-signed-certs\""
Oct 30 17:30:16.645144 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:16.645112    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq"]
Oct 30 17:30:16.689352 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:16.688633    4433 scope.go:122] "RemoveContainer" containerID="098fa0c59fc3f8e8336dffedbc72fcec39068df718ef603dc8f17a6a9be1e73a"
Oct 30 17:30:16.689352 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:16.688893    4433 scope.go:122] "RemoveContainer" containerID="cb3c1bbf7d378f6322609f3cdc6350e03d9eadd6dac9c6df6216f1ff244a04c9"
Oct 30 17:30:16.689633 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:16.689622    4433 scope.go:122] "RemoveContainer" containerID="61b24818b271e1f6b00f8e559b31886f0c08b22a46dd82444a65da1a5313c4bf"
Oct 30 17:30:16.759080 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:16.759050    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba03c7a7-48e4-4051-91d4-9a030790fa8a-dshm\") pod \"auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq\" (UID: \"ba03c7a7-48e4-4051-91d4-9a030790fa8a\") " pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq"
Oct 30 17:30:16.759573 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:16.759550    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba03c7a7-48e4-4051-91d4-9a030790fa8a-tmp-dir\") pod \"auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq\" (UID: \"ba03c7a7-48e4-4051-91d4-9a030790fa8a\") " pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq"
Oct 30 17:30:16.759799 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:16.759784    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba03c7a7-48e4-4051-91d4-9a030790fa8a-model-cache\") pod \"auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq\" (UID: \"ba03c7a7-48e4-4051-91d4-9a030790fa8a\") " pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq"
Oct 30 17:30:16.760137 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:16.760119    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba03c7a7-48e4-4051-91d4-9a030790fa8a-home\") pod \"auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq\" (UID: \"ba03c7a7-48e4-4051-91d4-9a030790fa8a\") " pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq"
Oct 30 17:30:16.760544 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:16.760526    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba03c7a7-48e4-4051-91d4-9a030790fa8a-tls-certs\") pod \"auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq\" (UID: \"ba03c7a7-48e4-4051-91d4-9a030790fa8a\") " pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq"
Oct 30 17:30:16.761552 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:16.761512    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cvcs\" (UniqueName: \"kubernetes.io/projected/ba03c7a7-48e4-4051-91d4-9a030790fa8a-kube-api-access-2cvcs\") pod \"auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq\" (UID: \"ba03c7a7-48e4-4051-91d4-9a030790fa8a\") " pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq"
Oct 30 17:30:16.785951 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:16.785904    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889690-wzlh5" podStartSLOduration=16.785890637 podStartE2EDuration="16.785890637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:16.781851049 +0000 UTC m=+2593.216753255" watchObservedRunningTime="2026-10-30 17:30:16.785890637 +0000 UTC m=+2593.220792838"
Oct 30 17:30:16.870442 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:16.868928    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba03c7a7-48e4-4051-91d4-9a030790fa8a-dshm\") pod \"auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq\" (UID: \"ba03c7a7-48e4-4051-91d4-9a030790fa8a\") " pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq"
Oct 30 17:30:16.870442 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:16.868983    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba03c7a7-48e4-4051-91d4-9a030790fa8a-tmp-dir\") pod \"auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq\" (UID: \"ba03c7a7-48e4-4051-91d4-9a030790fa8a\") " pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq"
Oct 30 17:30:16.870442 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:16.869066    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba03c7a7-48e4-4051-91d4-9a030790fa8a-model-cache\") pod \"auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq\" (UID: \"ba03c7a7-48e4-4051-91d4-9a030790fa8a\") " pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq"
Oct 30 17:30:16.870442 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:16.869122    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba03c7a7-48e4-4051-91d4-9a030790fa8a-home\") pod \"auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq\" (UID: \"ba03c7a7-48e4-4051-91d4-9a030790fa8a\") " pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq"
Oct 30 17:30:16.870442 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:16.869263    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba03c7a7-48e4-4051-91d4-9a030790fa8a-tls-certs\") pod \"auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq\" (UID: \"ba03c7a7-48e4-4051-91d4-9a030790fa8a\") " pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq"
Oct 30 17:30:16.870442 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:16.869371    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cvcs\" (UniqueName: \"kubernetes.io/projected/ba03c7a7-48e4-4051-91d4-9a030790fa8a-kube-api-access-2cvcs\") pod \"auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq\" (UID: \"ba03c7a7-48e4-4051-91d4-9a030790fa8a\") " pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq"
Oct 30 17:30:16.897440 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:16.871924    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba03c7a7-48e4-4051-91d4-9a030790fa8a-model-cache\") pod \"auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq\" (UID: \"ba03c7a7-48e4-4051-91d4-9a030790fa8a\") " pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq"
Oct 30 17:30:16.897440 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:16.872187    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba03c7a7-48e4-4051-91d4-9a030790fa8a-tmp-dir\") pod \"auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq\" (UID: \"ba03c7a7-48e4-4051-91d4-9a030790fa8a\") " pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq"
Oct 30 17:30:16.897440 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:16.872440    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba03c7a7-48e4-4051-91d4-9a030790fa8a-home\") pod \"auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq\" (UID: \"ba03c7a7-48e4-4051-91d4-9a030790fa8a\") " pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq"
Oct 30 17:30:16.908426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:16.898573    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba03c7a7-48e4-4051-91d4-9a030790fa8a-tls-certs\") pod \"auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq\" (UID: \"ba03c7a7-48e4-4051-91d4-9a030790fa8a\") " pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq"
Oct 30 17:30:16.930290 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:16.930221    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cvcs\" (UniqueName: \"kubernetes.io/projected/ba03c7a7-48e4-4051-91d4-9a030790fa8a-kube-api-access-2cvcs\") pod \"auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq\" (UID: \"ba03c7a7-48e4-4051-91d4-9a030790fa8a\") " pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq"
Oct 30 17:30:16.941968 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:16.941928    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba03c7a7-48e4-4051-91d4-9a030790fa8a-dshm\") pod \"auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq\" (UID: \"ba03c7a7-48e4-4051-91d4-9a030790fa8a\") " pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq"
Oct 30 17:30:17.236923 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.236890    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq"
Oct 30 17:30:17.508062 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.508037    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-f9c947b48-xn892"
Oct 30 17:30:17.548906 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.548823    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq"]
Oct 30 17:30:17.552531 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:30:17.552500    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba03c7a7_48e4_4051_91d4_9a030790fa8a.slice/crio-4a65f9cf9c102bc2562ca801f3a854b7e5732900de8f41e5fb23e68f58f091b4 WatchSource:0}: Error finding container 4a65f9cf9c102bc2562ca801f3a854b7e5732900de8f41e5fb23e68f58f091b4: Status 404 returned error can't find the container with id 4a65f9cf9c102bc2562ca801f3a854b7e5732900de8f41e5fb23e68f58f091b4
Oct 30 17:30:17.591289 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.591254    4433 generic.go:356] "Generic (PLEG): container finished" podID="c9d1683a-8e8a-4717-90ad-a4d84861d2f1" containerID="379a1235904a64c1c63a7fcddf9ac91e3345b431791f743016d0ca5eee7353f8" exitCode=60
Oct 30 17:30:17.591479 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.591352    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v" event={"ID":"c9d1683a-8e8a-4717-90ad-a4d84861d2f1","Type":"ContainerDied","Data":"379a1235904a64c1c63a7fcddf9ac91e3345b431791f743016d0ca5eee7353f8"}
Oct 30 17:30:17.591479 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.591463    4433 scope.go:122] "RemoveContainer" containerID="d6ec3e884af85d74a090b921c48ca562bbd5e47c3de80d2e019f0bbb693e7796"
Oct 30 17:30:17.592419 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.591928    4433 scope.go:122] "RemoveContainer" containerID="379a1235904a64c1c63a7fcddf9ac91e3345b431791f743016d0ca5eee7353f8"
Oct 30 17:30:17.592419 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:17.592255    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v_odh-ai-gateway-infra(c9d1683a-8e8a-4717-90ad-a4d84861d2f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v" podUID="c9d1683a-8e8a-4717-90ad-a4d84861d2f1"
Oct 30 17:30:17.596289 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.595687    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb" event={"ID":"35124e4a-306c-4d88-9321-d5de27812912","Type":"ContainerStarted","Data":"f9d84242bcbc2e217e3731b7e0e5fa1e79b7fb4185b0574e582ee025c5945fcf"}
Oct 30 17:30:17.598785 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.598766    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99" event={"ID":"73952f5d-729f-4433-aeeb-21c31b18c1d9","Type":"ContainerStarted","Data":"d56ea97274728a647cdb3749966276f9ec80e2d3cd826e896447ae012f186583"}
Oct 30 17:30:17.602314 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.602289    4433 generic.go:356] "Generic (PLEG): container finished" podID="993f683e-25cd-4f6e-beff-4da2e7c6ea69" containerID="c13587b489fd232e5de77f5e81f794d7789ce38f76b0e23be357776c34c08201" exitCode=60
Oct 30 17:30:17.602457 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.602373    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-5pb4x" event={"ID":"993f683e-25cd-4f6e-beff-4da2e7c6ea69","Type":"ContainerDied","Data":"c13587b489fd232e5de77f5e81f794d7789ce38f76b0e23be357776c34c08201"}
Oct 30 17:30:17.602750 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.602733    4433 scope.go:122] "RemoveContainer" containerID="c13587b489fd232e5de77f5e81f794d7789ce38f76b0e23be357776c34c08201"
Oct 30 17:30:17.603025 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:17.603003    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889690-5pb4x_odh-ai-gateway-infra(993f683e-25cd-4f6e-beff-4da2e7c6ea69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-5pb4x" podUID="993f683e-25cd-4f6e-beff-4da2e7c6ea69"
Oct 30 17:30:17.612106 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.611377    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj" event={"ID":"5d225e1b-96bc-4632-8088-60a96b84999c","Type":"ContainerStarted","Data":"be8fa91b7d9278250067c666855bb38fa5c90916dc89741188eb529f56ca99e1"}
Oct 30 17:30:17.618093 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.615889    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq" event={"ID":"ba03c7a7-48e4-4051-91d4-9a030790fa8a","Type":"ContainerStarted","Data":"4a65f9cf9c102bc2562ca801f3a854b7e5732900de8f41e5fb23e68f58f091b4"}
Oct 30 17:30:17.618764 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.618741    4433 generic.go:356] "Generic (PLEG): container finished" podID="786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3" containerID="7e8c2d2f28179df3ebd544949888e4fc9070edccb5473ce59c2a8941b0af8625" exitCode=60
Oct 30 17:30:17.618848 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.618803    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889690-wzlh5" event={"ID":"786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3","Type":"ContainerDied","Data":"7e8c2d2f28179df3ebd544949888e4fc9070edccb5473ce59c2a8941b0af8625"}
Oct 30 17:30:17.619167 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.619144    4433 scope.go:122] "RemoveContainer" containerID="7e8c2d2f28179df3ebd544949888e4fc9070edccb5473ce59c2a8941b0af8625"
Oct 30 17:30:17.619452 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:17.619428    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889690-wzlh5_odh-ai-gateway-infra(786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889690-wzlh5" podUID="786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3"
Oct 30 17:30:17.636969 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.627772    4433 generic.go:356] "Generic (PLEG): container finished" podID="12e09dc3-bfd6-4b8b-90ca-8fab73d0c922" containerID="b1a18f7cb731a67f023afd08a39ca7d4f5cd681fbab087566d78ad47251079a7" exitCode=60
Oct 30 17:30:17.636969 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.628437    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf" event={"ID":"12e09dc3-bfd6-4b8b-90ca-8fab73d0c922","Type":"ContainerDied","Data":"b1a18f7cb731a67f023afd08a39ca7d4f5cd681fbab087566d78ad47251079a7"}
Oct 30 17:30:17.636969 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.629600    4433 scope.go:122] "RemoveContainer" containerID="b1a18f7cb731a67f023afd08a39ca7d4f5cd681fbab087566d78ad47251079a7"
Oct 30 17:30:17.636969 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:17.630738    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf_odh-ai-gateway-infra(12e09dc3-bfd6-4b8b-90ca-8fab73d0c922)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf" podUID="12e09dc3-bfd6-4b8b-90ca-8fab73d0c922"
Oct 30 17:30:17.636969 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.631326    4433 scope.go:122] "RemoveContainer" containerID="e10a5fc36b8194e37be5063bb19bcb3a6b72199bbb9d15fbbabed9ef9d833090"
Oct 30 17:30:17.636969 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.635791    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v"]
Oct 30 17:30:17.651763 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.651728    4433 generic.go:356] "Generic (PLEG): container finished" podID="db5021f4-7307-4efa-8155-dc88fb2d95de" containerID="8029cea349b7e154700b07f83da20617d7bb3e3c2767ac42d923e2e914799bc3" exitCode=60
Oct 30 17:30:17.651942 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.651829    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m" event={"ID":"db5021f4-7307-4efa-8155-dc88fb2d95de","Type":"ContainerDied","Data":"8029cea349b7e154700b07f83da20617d7bb3e3c2767ac42d923e2e914799bc3"}
Oct 30 17:30:17.657925 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.657900    4433 scope.go:122] "RemoveContainer" containerID="8029cea349b7e154700b07f83da20617d7bb3e3c2767ac42d923e2e914799bc3"
Oct 30 17:30:17.658700 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:17.658674    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m_odh-ai-gateway-infra(db5021f4-7307-4efa-8155-dc88fb2d95de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m" podUID="db5021f4-7307-4efa-8155-dc88fb2d95de"
Oct 30 17:30:17.676673 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.676637    4433 scope.go:122] "RemoveContainer" containerID="0b7a652e52157980502a0539ea2a9b3c2d56c784ac8dcf17bc7c6f523a1f809e"
Oct 30 17:30:17.680514 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.680473    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m"]
Oct 30 17:30:17.690557 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.688614    4433 scope.go:122] "RemoveContainer" containerID="d03dbb888e331a91797ddac825826477a5311d5ac520e3676e10f71731dc79c8"
Oct 30 17:30:17.690557 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.689262    4433 scope.go:122] "RemoveContainer" containerID="41cfb757fbb61c0d546efc6771ecce3f86b38a27cb2c4ee4903ca8a5c9709dec"
Oct 30 17:30:17.690557 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.689367    4433 scope.go:122] "RemoveContainer" containerID="e682c9bef44ab1f46c74ce5db0941719bb488ef6dc902060c0ec2be72df48d3e"
Oct 30 17:30:17.690928 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.690601    4433 scope.go:122] "RemoveContainer" containerID="b6e139cfe9fb0aabb2ea36d6ac7e2be78a2d5681fe38e6fd9bd1dcec72f84134"
Oct 30 17:30:17.708864 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.708839    4433 scope.go:122] "RemoveContainer" containerID="f4f38d3569f383579f89c799ac636b53e8445561645dd675fb5acd275e5bf48b"
Oct 30 17:30:17.717294 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.717259    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889690-5pb4x"]
Oct 30 17:30:17.761304 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.761275    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf"]
Oct 30 17:30:17.797038 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.797001    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889690-wzlh5"]
Oct 30 17:30:17.859783 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:17.854630    4433 scope.go:122] "RemoveContainer" containerID="850362c6459442a22f5647c95f834c9cf172347affd4d8ca6745854a5b8e3ad9"
Oct 30 17:30:18.663266 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:18.663237    4433 generic.go:356] "Generic (PLEG): container finished" podID="35124e4a-306c-4d88-9321-d5de27812912" containerID="f9d84242bcbc2e217e3731b7e0e5fa1e79b7fb4185b0574e582ee025c5945fcf" exitCode=60
Oct 30 17:30:18.663431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:18.663327    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb" event={"ID":"35124e4a-306c-4d88-9321-d5de27812912","Type":"ContainerDied","Data":"f9d84242bcbc2e217e3731b7e0e5fa1e79b7fb4185b0574e582ee025c5945fcf"}
Oct 30 17:30:18.663431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:18.663416    4433 scope.go:122] "RemoveContainer" containerID="cb3c1bbf7d378f6322609f3cdc6350e03d9eadd6dac9c6df6216f1ff244a04c9"
Oct 30 17:30:18.664061 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:18.664041    4433 scope.go:122] "RemoveContainer" containerID="f9d84242bcbc2e217e3731b7e0e5fa1e79b7fb4185b0574e582ee025c5945fcf"
Oct 30 17:30:18.664503 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:18.664478    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb_odh-ai-gateway-infra(35124e4a-306c-4d88-9321-d5de27812912)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb" podUID="35124e4a-306c-4d88-9321-d5de27812912"
Oct 30 17:30:18.669160 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:18.669137    4433 generic.go:356] "Generic (PLEG): container finished" podID="73952f5d-729f-4433-aeeb-21c31b18c1d9" containerID="d56ea97274728a647cdb3749966276f9ec80e2d3cd826e896447ae012f186583" exitCode=60
Oct 30 17:30:18.669261 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:18.669221    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99" event={"ID":"73952f5d-729f-4433-aeeb-21c31b18c1d9","Type":"ContainerDied","Data":"d56ea97274728a647cdb3749966276f9ec80e2d3cd826e896447ae012f186583"}
Oct 30 17:30:18.669795 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:18.669774    4433 scope.go:122] "RemoveContainer" containerID="d56ea97274728a647cdb3749966276f9ec80e2d3cd826e896447ae012f186583"
Oct 30 17:30:18.670248 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:18.670206    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99_odh-ai-gateway-infra(73952f5d-729f-4433-aeeb-21c31b18c1d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99" podUID="73952f5d-729f-4433-aeeb-21c31b18c1d9"
Oct 30 17:30:18.673334 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:18.673313    4433 generic.go:356] "Generic (PLEG): container finished" podID="5d225e1b-96bc-4632-8088-60a96b84999c" containerID="be8fa91b7d9278250067c666855bb38fa5c90916dc89741188eb529f56ca99e1" exitCode=60
Oct 30 17:30:18.673468 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:18.673391    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj" event={"ID":"5d225e1b-96bc-4632-8088-60a96b84999c","Type":"ContainerDied","Data":"be8fa91b7d9278250067c666855bb38fa5c90916dc89741188eb529f56ca99e1"}
Oct 30 17:30:18.673985 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:18.673964    4433 scope.go:122] "RemoveContainer" containerID="be8fa91b7d9278250067c666855bb38fa5c90916dc89741188eb529f56ca99e1"
Oct 30 17:30:18.674378 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:18.674343    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj_odh-ai-gateway-infra(5d225e1b-96bc-4632-8088-60a96b84999c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj" podUID="5d225e1b-96bc-4632-8088-60a96b84999c"
Oct 30 17:30:18.675656 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:18.675610    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq" event={"ID":"ba03c7a7-48e4-4051-91d4-9a030790fa8a","Type":"ContainerStarted","Data":"169568da9f2606562778832bc83a39b45d409e21d589a2bdbf5c58be348b3c54"}
Oct 30 17:30:18.678715 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:18.678688    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg" event={"ID":"f02fcdb6-55c4-4713-b726-ff1de3d3a7ba","Type":"ContainerStarted","Data":"f1b988bb7536457f43e09de6d1a06e77261f571b7c52d4fe1cdf65555a0ab929"}
Oct 30 17:30:18.683145 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:18.683118    4433 scope.go:122] "RemoveContainer" containerID="61b24818b271e1f6b00f8e559b31886f0c08b22a46dd82444a65da1a5313c4bf"
Oct 30 17:30:18.694421 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:18.694382    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws" event={"ID":"c3961d3d-0ada-44af-a530-2a7565e96257","Type":"ContainerStarted","Data":"ff85e04bd71efd87bd1992ab15a4c5184c5bc001c775a8999cc1d7d96c69fcb0"}
Oct 30 17:30:18.695892 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:18.695855    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj"]
Oct 30 17:30:18.698036 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:18.697906    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl" event={"ID":"5668479f-566c-4d0a-9353-7a60f275673a","Type":"ContainerStarted","Data":"827245adc9428d6c4ef9ae3b1f405f58883fd04e46d93d181f7c1fa745063c62"}
Oct 30 17:30:18.702582 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:18.702001    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n" event={"ID":"b01cc039-2221-4716-8611-8d08004e4778","Type":"ContainerStarted","Data":"333b3ac01545205fa32a4cbf64f3fb9890c28048cafe9ca3af1df0aaa4670cb1"}
Oct 30 17:30:18.742537 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:18.742201    4433 scope.go:122] "RemoveContainer" containerID="098fa0c59fc3f8e8336dffedbc72fcec39068df718ef603dc8f17a6a9be1e73a"
Oct 30 17:30:18.774684 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:18.774643    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99"]
Oct 30 17:30:18.803205 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:18.803118    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb"]
Oct 30 17:30:18.916296 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:18.916230    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq" podStartSLOduration=2.916207972 podStartE2EDuration="2.916207972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:18.860779223 +0000 UTC m=+2595.295681429" watchObservedRunningTime="2026-10-30 17:30:18.916207972 +0000 UTC m=+2595.351110178"
Oct 30 17:30:18.924192 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:18.924111    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v"
Oct 30 17:30:19.059503 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.050701    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9d1683a-8e8a-4717-90ad-a4d84861d2f1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9d1683a-8e8a-4717-90ad-a4d84861d2f1-maas-api-service-ca\") pod \"c9d1683a-8e8a-4717-90ad-a4d84861d2f1\" (UID: \"c9d1683a-8e8a-4717-90ad-a4d84861d2f1\") "
Oct 30 17:30:19.059503 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.051146    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9d1683a-8e8a-4717-90ad-a4d84861d2f1-kube-api-access-s7drh\" (UniqueName: \"kubernetes.io/projected/c9d1683a-8e8a-4717-90ad-a4d84861d2f1-kube-api-access-s7drh\") pod \"c9d1683a-8e8a-4717-90ad-a4d84861d2f1\" (UID: \"c9d1683a-8e8a-4717-90ad-a4d84861d2f1\") "
Oct 30 17:30:19.059503 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.056878    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9d1683a-8e8a-4717-90ad-a4d84861d2f1-kube-api-access-s7drh" pod "c9d1683a-8e8a-4717-90ad-a4d84861d2f1" (UID: "c9d1683a-8e8a-4717-90ad-a4d84861d2f1"). InnerVolumeSpecName "kube-api-access-s7drh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:30:19.061591 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.061562    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9d1683a-8e8a-4717-90ad-a4d84861d2f1-maas-api-service-ca" pod "c9d1683a-8e8a-4717-90ad-a4d84861d2f1" (UID: "c9d1683a-8e8a-4717-90ad-a4d84861d2f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:30:19.158271 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.158169    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7drh\" (UniqueName: \"kubernetes.io/projected/c9d1683a-8e8a-4717-90ad-a4d84861d2f1-kube-api-access-s7drh\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:30:19.158271 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.158202    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9d1683a-8e8a-4717-90ad-a4d84861d2f1-maas-api-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:30:19.241023 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.240963    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq"
Oct 30 17:30:19.271823 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.269695    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889690-wzlh5"
Oct 30 17:30:19.364037 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.363961    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf"
Oct 30 17:30:19.367078 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.367032    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3-kube-api-access-gt87z\" (UniqueName: \"kubernetes.io/projected/786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3-kube-api-access-gt87z\") pod \"786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3\" (UID: \"786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3\") "
Oct 30 17:30:19.371088 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.370908    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3-maas-api-service-ca\") pod \"786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3\" (UID: \"786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3\") "
Oct 30 17:30:19.371353 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.371327    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3-maas-api-service-ca" pod "786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3" (UID: "786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:30:19.372300 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.372267    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3-maas-api-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:30:19.373172 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.373150    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-5pb4x"
Oct 30 17:30:19.376901 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.376669    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m"
Oct 30 17:30:19.377532 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.377506    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3-kube-api-access-gt87z" pod "786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3" (UID: "786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3"). InnerVolumeSpecName "kube-api-access-gt87z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:30:19.482428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.473741    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db5021f4-7307-4efa-8155-dc88fb2d95de-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db5021f4-7307-4efa-8155-dc88fb2d95de-maas-api-service-ca\") pod \"db5021f4-7307-4efa-8155-dc88fb2d95de\" (UID: \"db5021f4-7307-4efa-8155-dc88fb2d95de\") "
Oct 30 17:30:19.482428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.473838    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12e09dc3-bfd6-4b8b-90ca-8fab73d0c922-kube-api-access-gqtp9\" (UniqueName: \"kubernetes.io/projected/12e09dc3-bfd6-4b8b-90ca-8fab73d0c922-kube-api-access-gqtp9\") pod \"12e09dc3-bfd6-4b8b-90ca-8fab73d0c922\" (UID: \"12e09dc3-bfd6-4b8b-90ca-8fab73d0c922\") "
Oct 30 17:30:19.482428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.473897    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12e09dc3-bfd6-4b8b-90ca-8fab73d0c922-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12e09dc3-bfd6-4b8b-90ca-8fab73d0c922-maas-api-service-ca\") pod \"12e09dc3-bfd6-4b8b-90ca-8fab73d0c922\" (UID: \"12e09dc3-bfd6-4b8b-90ca-8fab73d0c922\") "
Oct 30 17:30:19.482428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.474078    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db5021f4-7307-4efa-8155-dc88fb2d95de-kube-api-access-9w9d9\" (UniqueName: \"kubernetes.io/projected/db5021f4-7307-4efa-8155-dc88fb2d95de-kube-api-access-9w9d9\") pod \"db5021f4-7307-4efa-8155-dc88fb2d95de\" (UID: \"db5021f4-7307-4efa-8155-dc88fb2d95de\") "
Oct 30 17:30:19.482428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.474205    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/993f683e-25cd-4f6e-beff-4da2e7c6ea69-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/993f683e-25cd-4f6e-beff-4da2e7c6ea69-maas-api-service-ca\") pod \"993f683e-25cd-4f6e-beff-4da2e7c6ea69\" (UID: \"993f683e-25cd-4f6e-beff-4da2e7c6ea69\") "
Oct 30 17:30:19.482428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.474241    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/993f683e-25cd-4f6e-beff-4da2e7c6ea69-kube-api-access-cphvf\" (UniqueName: \"kubernetes.io/projected/993f683e-25cd-4f6e-beff-4da2e7c6ea69-kube-api-access-cphvf\") pod \"993f683e-25cd-4f6e-beff-4da2e7c6ea69\" (UID: \"993f683e-25cd-4f6e-beff-4da2e7c6ea69\") "
Oct 30 17:30:19.482428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.474933    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt87z\" (UniqueName: \"kubernetes.io/projected/786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3-kube-api-access-gt87z\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:30:19.482428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.478285    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/993f683e-25cd-4f6e-beff-4da2e7c6ea69-kube-api-access-cphvf" pod "993f683e-25cd-4f6e-beff-4da2e7c6ea69" (UID: "993f683e-25cd-4f6e-beff-4da2e7c6ea69"). InnerVolumeSpecName "kube-api-access-cphvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:30:19.482428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.478650    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/993f683e-25cd-4f6e-beff-4da2e7c6ea69-maas-api-service-ca" pod "993f683e-25cd-4f6e-beff-4da2e7c6ea69" (UID: "993f683e-25cd-4f6e-beff-4da2e7c6ea69"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:30:19.482428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.478986    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db5021f4-7307-4efa-8155-dc88fb2d95de-maas-api-service-ca" pod "db5021f4-7307-4efa-8155-dc88fb2d95de" (UID: "db5021f4-7307-4efa-8155-dc88fb2d95de"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:30:19.482428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.481934    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db5021f4-7307-4efa-8155-dc88fb2d95de-kube-api-access-9w9d9" pod "db5021f4-7307-4efa-8155-dc88fb2d95de" (UID: "db5021f4-7307-4efa-8155-dc88fb2d95de"). InnerVolumeSpecName "kube-api-access-9w9d9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:30:19.483148 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.482610    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12e09dc3-bfd6-4b8b-90ca-8fab73d0c922-maas-api-service-ca" pod "12e09dc3-bfd6-4b8b-90ca-8fab73d0c922" (UID: "12e09dc3-bfd6-4b8b-90ca-8fab73d0c922"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:30:19.488427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.483952    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12e09dc3-bfd6-4b8b-90ca-8fab73d0c922-kube-api-access-gqtp9" pod "12e09dc3-bfd6-4b8b-90ca-8fab73d0c922" (UID: "12e09dc3-bfd6-4b8b-90ca-8fab73d0c922"). InnerVolumeSpecName "kube-api-access-gqtp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:30:19.607078 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.606928    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w9d9\" (UniqueName: \"kubernetes.io/projected/db5021f4-7307-4efa-8155-dc88fb2d95de-kube-api-access-9w9d9\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:30:19.607078 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.606960    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/993f683e-25cd-4f6e-beff-4da2e7c6ea69-maas-api-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:30:19.607078 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.606977    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cphvf\" (UniqueName: \"kubernetes.io/projected/993f683e-25cd-4f6e-beff-4da2e7c6ea69-kube-api-access-cphvf\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:30:19.607078 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.606990    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db5021f4-7307-4efa-8155-dc88fb2d95de-maas-api-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:30:19.607078 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.607003    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqtp9\" (UniqueName: \"kubernetes.io/projected/12e09dc3-bfd6-4b8b-90ca-8fab73d0c922-kube-api-access-gqtp9\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:30:19.607078 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.607017    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12e09dc3-bfd6-4b8b-90ca-8fab73d0c922-maas-api-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:30:19.733657 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.733621    4433 generic.go:356] "Generic (PLEG): container finished" podID="b01cc039-2221-4716-8611-8d08004e4778" containerID="333b3ac01545205fa32a4cbf64f3fb9890c28048cafe9ca3af1df0aaa4670cb1" exitCode=60
Oct 30 17:30:19.841473 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.841126    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n" event={"ID":"b01cc039-2221-4716-8611-8d08004e4778","Type":"ContainerDied","Data":"333b3ac01545205fa32a4cbf64f3fb9890c28048cafe9ca3af1df0aaa4670cb1"}
Oct 30 17:30:19.841473 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.841233    4433 scope.go:122] "RemoveContainer" containerID="d03dbb888e331a91797ddac825826477a5311d5ac520e3676e10f71731dc79c8"
Oct 30 17:30:19.857944 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.857912    4433 scope.go:122] "RemoveContainer" containerID="333b3ac01545205fa32a4cbf64f3fb9890c28048cafe9ca3af1df0aaa4670cb1"
Oct 30 17:30:19.858668 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:19.858636    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n_odh-ai-gateway-infra(b01cc039-2221-4716-8611-8d08004e4778)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n" podUID="b01cc039-2221-4716-8611-8d08004e4778"
Oct 30 17:30:19.892003 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.891976    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v"
Oct 30 17:30:19.901769 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.892991    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v" event={"ID":"c9d1683a-8e8a-4717-90ad-a4d84861d2f1","Type":"ContainerDied","Data":"9b2692fb27449882c5481bd627af66bcc63413be87b2714e0c098e01166a05ba"}
Oct 30 17:30:19.914254 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.908712    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-5pb4x" event={"ID":"993f683e-25cd-4f6e-beff-4da2e7c6ea69","Type":"ContainerDied","Data":"8cb938c4f8b808b73688e7f71292e14192bac3dd5e11577c260d9266d139c5ee"}
Oct 30 17:30:19.914254 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.908826    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-5pb4x"
Oct 30 17:30:19.932006 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.931977    4433 generic.go:356] "Generic (PLEG): container finished" podID="f02fcdb6-55c4-4713-b726-ff1de3d3a7ba" containerID="f1b988bb7536457f43e09de6d1a06e77261f571b7c52d4fe1cdf65555a0ab929" exitCode=60
Oct 30 17:30:19.932494 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.932275    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg" event={"ID":"f02fcdb6-55c4-4713-b726-ff1de3d3a7ba","Type":"ContainerDied","Data":"f1b988bb7536457f43e09de6d1a06e77261f571b7c52d4fe1cdf65555a0ab929"}
Oct 30 17:30:19.946068 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.946039    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889690-wzlh5" event={"ID":"786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3","Type":"ContainerDied","Data":"218cc0907bd8f2152edb07ab535c938c728fd6cca672a5f7ce825854d86c48e4"}
Oct 30 17:30:19.946893 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.946858    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf" event={"ID":"12e09dc3-bfd6-4b8b-90ca-8fab73d0c922","Type":"ContainerDied","Data":"5160b344f4158ae3ad0f2a0d0ad899c6f6cfeb960c890037c57fa9af18f28aac"}
Oct 30 17:30:19.947191 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.947066    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m" event={"ID":"db5021f4-7307-4efa-8155-dc88fb2d95de","Type":"ContainerDied","Data":"c3d163290bb01701a8b80ffb240c002089e09b43171378a906a9459dc9e1d1ff"}
Oct 30 17:30:19.947191 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.946808    4433 scope.go:122] "RemoveContainer" containerID="f1b988bb7536457f43e09de6d1a06e77261f571b7c52d4fe1cdf65555a0ab929"
Oct 30 17:30:19.951230 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.934455    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889690-wzlh5"
Oct 30 17:30:19.951230 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.936311    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf"
Oct 30 17:30:19.951230 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.942603    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m"
Oct 30 17:30:19.952040 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:19.951838    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg_odh-ai-gateway-infra(f02fcdb6-55c4-4713-b726-ff1de3d3a7ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg" podUID="f02fcdb6-55c4-4713-b726-ff1de3d3a7ba"
Oct 30 17:30:19.977417 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.977328    4433 scope.go:122] "RemoveContainer" containerID="379a1235904a64c1c63a7fcddf9ac91e3345b431791f743016d0ca5eee7353f8"
Oct 30 17:30:19.983999 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.982614    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg"]
Oct 30 17:30:19.983999 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.983157    4433 generic.go:356] "Generic (PLEG): container finished" podID="c3961d3d-0ada-44af-a530-2a7565e96257" containerID="ff85e04bd71efd87bd1992ab15a4c5184c5bc001c775a8999cc1d7d96c69fcb0" exitCode=60
Oct 30 17:30:19.983999 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.983318    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws" event={"ID":"c3961d3d-0ada-44af-a530-2a7565e96257","Type":"ContainerDied","Data":"ff85e04bd71efd87bd1992ab15a4c5184c5bc001c775a8999cc1d7d96c69fcb0"}
Oct 30 17:30:19.983999 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.983910    4433 scope.go:122] "RemoveContainer" containerID="ff85e04bd71efd87bd1992ab15a4c5184c5bc001c775a8999cc1d7d96c69fcb0"
Oct 30 17:30:19.985099 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:19.984342    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws_odh-ai-gateway-infra(c3961d3d-0ada-44af-a530-2a7565e96257)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws" podUID="c3961d3d-0ada-44af-a530-2a7565e96257"
Oct 30 17:30:19.996502 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.987507    4433 generic.go:356] "Generic (PLEG): container finished" podID="5668479f-566c-4d0a-9353-7a60f275673a" containerID="827245adc9428d6c4ef9ae3b1f405f58883fd04e46d93d181f7c1fa745063c62" exitCode=60
Oct 30 17:30:19.996502 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.988417    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl" event={"ID":"5668479f-566c-4d0a-9353-7a60f275673a","Type":"ContainerDied","Data":"827245adc9428d6c4ef9ae3b1f405f58883fd04e46d93d181f7c1fa745063c62"}
Oct 30 17:30:19.996502 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:19.988812    4433 scope.go:122] "RemoveContainer" containerID="827245adc9428d6c4ef9ae3b1f405f58883fd04e46d93d181f7c1fa745063c62"
Oct 30 17:30:19.996502 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:19.989149    4433 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl_odh-ai-gateway-infra(5668479f-566c-4d0a-9353-7a60f275673a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl" podUID="5668479f-566c-4d0a-9353-7a60f275673a"
Oct 30 17:30:20.010292 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.010263    4433 scope.go:122] "RemoveContainer" containerID="c13587b489fd232e5de77f5e81f794d7789ce38f76b0e23be357776c34c08201"
Oct 30 17:30:20.019469 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.011681    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws"]
Oct 30 17:30:20.033728 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.033192    4433 scope.go:122] "RemoveContainer" containerID="b6e139cfe9fb0aabb2ea36d6ac7e2be78a2d5681fe38e6fd9bd1dcec72f84134"
Oct 30 17:30:20.033821 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.033781    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl"]
Oct 30 17:30:20.061612 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.061580    4433 scope.go:122] "RemoveContainer" containerID="7e8c2d2f28179df3ebd544949888e4fc9070edccb5473ce59c2a8941b0af8625"
Oct 30 17:30:20.092821 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.092790    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889690-5pb4x"]
Oct 30 17:30:20.098722 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.098677    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889690-5pb4x"]
Oct 30 17:30:20.100244 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.100135    4433 scope.go:122] "RemoveContainer" containerID="b1a18f7cb731a67f023afd08a39ca7d4f5cd681fbab087566d78ad47251079a7"
Oct 30 17:30:20.115077 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.114983    4433 scope.go:122] "RemoveContainer" containerID="8029cea349b7e154700b07f83da20617d7bb3e3c2767ac42d923e2e914799bc3"
Oct 30 17:30:20.115376 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.115355    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n"]
Oct 30 17:30:20.131567 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.128263    4433 scope.go:122] "RemoveContainer" containerID="41cfb757fbb61c0d546efc6771ecce3f86b38a27cb2c4ee4903ca8a5c9709dec"
Oct 30 17:30:20.136391 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.136339    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf"]
Oct 30 17:30:20.150463 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.148808    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a43843-29889690-r72jf"]
Oct 30 17:30:20.155794 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.155769    4433 scope.go:122] "RemoveContainer" containerID="e682c9bef44ab1f46c74ce5db0941719bb488ef6dc902060c0ec2be72df48d3e"
Oct 30 17:30:20.194684 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.194656    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889690-wzlh5"]
Oct 30 17:30:20.198181 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.198158    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-78c9c2-29889690-wzlh5"]
Oct 30 17:30:20.257507 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.256741    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb"
Oct 30 17:30:20.279277 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.279232    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v"]
Oct 30 17:30:20.289295 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.289260    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-8a7e4a-29889690-9fv6v"]
Oct 30 17:30:20.341037 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.341006    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35124e4a-306c-4d88-9321-d5de27812912-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35124e4a-306c-4d88-9321-d5de27812912-maas-api-service-ca\") pod \"35124e4a-306c-4d88-9321-d5de27812912\" (UID: \"35124e4a-306c-4d88-9321-d5de27812912\") "
Oct 30 17:30:20.341264 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.341241    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35124e4a-306c-4d88-9321-d5de27812912-kube-api-access-z5dzr\" (UniqueName: \"kubernetes.io/projected/35124e4a-306c-4d88-9321-d5de27812912-kube-api-access-z5dzr\") pod \"35124e4a-306c-4d88-9321-d5de27812912\" (UID: \"35124e4a-306c-4d88-9321-d5de27812912\") "
Oct 30 17:30:20.341441 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.341398    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35124e4a-306c-4d88-9321-d5de27812912-maas-api-service-ca" pod "35124e4a-306c-4d88-9321-d5de27812912" (UID: "35124e4a-306c-4d88-9321-d5de27812912"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:30:20.341913 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.341891    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35124e4a-306c-4d88-9321-d5de27812912-maas-api-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:30:20.343538 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.343511    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35124e4a-306c-4d88-9321-d5de27812912-kube-api-access-z5dzr" pod "35124e4a-306c-4d88-9321-d5de27812912" (UID: "35124e4a-306c-4d88-9321-d5de27812912"). InnerVolumeSpecName "kube-api-access-z5dzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:30:20.348190 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.348169    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj"
Oct 30 17:30:20.351229 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.351210    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99"
Oct 30 17:30:20.442918 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.442883    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73952f5d-729f-4433-aeeb-21c31b18c1d9-kube-api-access-dqs75\" (UniqueName: \"kubernetes.io/projected/73952f5d-729f-4433-aeeb-21c31b18c1d9-kube-api-access-dqs75\") pod \"73952f5d-729f-4433-aeeb-21c31b18c1d9\" (UID: \"73952f5d-729f-4433-aeeb-21c31b18c1d9\") "
Oct 30 17:30:20.443104 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.442967    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d225e1b-96bc-4632-8088-60a96b84999c-kube-api-access-nbrcz\" (UniqueName: \"kubernetes.io/projected/5d225e1b-96bc-4632-8088-60a96b84999c-kube-api-access-nbrcz\") pod \"5d225e1b-96bc-4632-8088-60a96b84999c\" (UID: \"5d225e1b-96bc-4632-8088-60a96b84999c\") "
Oct 30 17:30:20.443104 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.443063    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d225e1b-96bc-4632-8088-60a96b84999c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d225e1b-96bc-4632-8088-60a96b84999c-maas-api-service-ca\") pod \"5d225e1b-96bc-4632-8088-60a96b84999c\" (UID: \"5d225e1b-96bc-4632-8088-60a96b84999c\") "
Oct 30 17:30:20.443244 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.443213    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73952f5d-729f-4433-aeeb-21c31b18c1d9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73952f5d-729f-4433-aeeb-21c31b18c1d9-maas-api-service-ca\") pod \"73952f5d-729f-4433-aeeb-21c31b18c1d9\" (UID: \"73952f5d-729f-4433-aeeb-21c31b18c1d9\") "
Oct 30 17:30:20.443616 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.443579    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d225e1b-96bc-4632-8088-60a96b84999c-maas-api-service-ca" pod "5d225e1b-96bc-4632-8088-60a96b84999c" (UID: "5d225e1b-96bc-4632-8088-60a96b84999c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:30:20.443800 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.443732    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73952f5d-729f-4433-aeeb-21c31b18c1d9-maas-api-service-ca" pod "73952f5d-729f-4433-aeeb-21c31b18c1d9" (UID: "73952f5d-729f-4433-aeeb-21c31b18c1d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:30:20.444249 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.444223    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73952f5d-729f-4433-aeeb-21c31b18c1d9-maas-api-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:30:20.444348 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.444254    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5dzr\" (UniqueName: \"kubernetes.io/projected/35124e4a-306c-4d88-9321-d5de27812912-kube-api-access-z5dzr\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:30:20.444348 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.444268    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d225e1b-96bc-4632-8088-60a96b84999c-maas-api-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:30:20.445513 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.445490    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d225e1b-96bc-4632-8088-60a96b84999c-kube-api-access-nbrcz" pod "5d225e1b-96bc-4632-8088-60a96b84999c" (UID: "5d225e1b-96bc-4632-8088-60a96b84999c"). InnerVolumeSpecName "kube-api-access-nbrcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:30:20.445704 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.445682    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73952f5d-729f-4433-aeeb-21c31b18c1d9-kube-api-access-dqs75" pod "73952f5d-729f-4433-aeeb-21c31b18c1d9" (UID: "73952f5d-729f-4433-aeeb-21c31b18c1d9"). InnerVolumeSpecName "kube-api-access-dqs75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:30:20.449898 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.449871    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m"]
Oct 30 17:30:20.453683 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.453658    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75586d-29889690-m6k7m"]
Oct 30 17:30:20.551508 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.549291    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbrcz\" (UniqueName: \"kubernetes.io/projected/5d225e1b-96bc-4632-8088-60a96b84999c-kube-api-access-nbrcz\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:30:20.551508 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.549330    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqs75\" (UniqueName: \"kubernetes.io/projected/73952f5d-729f-4433-aeeb-21c31b18c1d9-kube-api-access-dqs75\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:30:20.996596 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.996555    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb" event={"ID":"35124e4a-306c-4d88-9321-d5de27812912","Type":"ContainerDied","Data":"494b7ccc077ce6566ae34871bc177aad1e83b5456f2c32cb8d9ebd8a34437dbb"}
Oct 30 17:30:20.996596 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.996590    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb"
Oct 30 17:30:20.997075 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.996627    4433 scope.go:122] "RemoveContainer" containerID="f9d84242bcbc2e217e3731b7e0e5fa1e79b7fb4185b0574e582ee025c5945fcf"
Oct 30 17:30:20.998791 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.998764    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99"
Oct 30 17:30:20.998791 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:20.998773    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99" event={"ID":"73952f5d-729f-4433-aeeb-21c31b18c1d9","Type":"ContainerDied","Data":"b65ab4af1fd0d9d782b0d8376f7db8e3e1a3c23fd493397f2b690be5cd477d64"}
Oct 30 17:30:21.002207 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.001916    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj" event={"ID":"5d225e1b-96bc-4632-8088-60a96b84999c","Type":"ContainerDied","Data":"c7388bcbe8510cdd8371c23e4f23e19a98b670027d204473323cb4566c7a0ea5"}
Oct 30 17:30:21.002207 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.001924    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj"
Oct 30 17:30:21.022930 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.022828    4433 scope.go:122] "RemoveContainer" containerID="d56ea97274728a647cdb3749966276f9ec80e2d3cd826e896447ae012f186583"
Oct 30 17:30:21.041203 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.039709    4433 scope.go:122] "RemoveContainer" containerID="be8fa91b7d9278250067c666855bb38fa5c90916dc89741188eb529f56ca99e1"
Oct 30 17:30:21.150666 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.150629    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj"]
Oct 30 17:30:21.157497 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.157438    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c7b9df-29889690-fxrdj"]
Oct 30 17:30:21.239047 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.239019    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg"
Oct 30 17:30:21.268476 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.268365    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99"]
Oct 30 17:30:21.277437 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.277388    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-11f480-29889690-s9n99"]
Oct 30 17:30:21.350112 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.350080    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb"]
Oct 30 17:30:21.357054 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.357024    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n"
Oct 30 17:30:21.359474 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.359453    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-86f50a-29889690-qh9sb"]
Oct 30 17:30:21.361363 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.361344    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl"
Oct 30 17:30:21.366539 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.366517    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws"
Oct 30 17:30:21.382853 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.382824    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f02fcdb6-55c4-4713-b726-ff1de3d3a7ba-kube-api-access-l45sh\" (UniqueName: \"kubernetes.io/projected/f02fcdb6-55c4-4713-b726-ff1de3d3a7ba-kube-api-access-l45sh\") pod \"f02fcdb6-55c4-4713-b726-ff1de3d3a7ba\" (UID: \"f02fcdb6-55c4-4713-b726-ff1de3d3a7ba\") "
Oct 30 17:30:21.382985 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.382936    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f02fcdb6-55c4-4713-b726-ff1de3d3a7ba-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f02fcdb6-55c4-4713-b726-ff1de3d3a7ba-maas-api-service-ca\") pod \"f02fcdb6-55c4-4713-b726-ff1de3d3a7ba\" (UID: \"f02fcdb6-55c4-4713-b726-ff1de3d3a7ba\") "
Oct 30 17:30:21.383280 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.383250    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f02fcdb6-55c4-4713-b726-ff1de3d3a7ba-maas-api-service-ca" pod "f02fcdb6-55c4-4713-b726-ff1de3d3a7ba" (UID: "f02fcdb6-55c4-4713-b726-ff1de3d3a7ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:30:21.384102 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.384043    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f02fcdb6-55c4-4713-b726-ff1de3d3a7ba-maas-api-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:30:21.385245 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.385221    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f02fcdb6-55c4-4713-b726-ff1de3d3a7ba-kube-api-access-l45sh" pod "f02fcdb6-55c4-4713-b726-ff1de3d3a7ba" (UID: "f02fcdb6-55c4-4713-b726-ff1de3d3a7ba"). InnerVolumeSpecName "kube-api-access-l45sh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:30:21.487749 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.486771    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b01cc039-2221-4716-8611-8d08004e4778-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b01cc039-2221-4716-8611-8d08004e4778-maas-api-service-ca\") pod \"b01cc039-2221-4716-8611-8d08004e4778\" (UID: \"b01cc039-2221-4716-8611-8d08004e4778\") "
Oct 30 17:30:21.487749 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.486843    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3961d3d-0ada-44af-a530-2a7565e96257-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3961d3d-0ada-44af-a530-2a7565e96257-maas-api-service-ca\") pod \"c3961d3d-0ada-44af-a530-2a7565e96257\" (UID: \"c3961d3d-0ada-44af-a530-2a7565e96257\") "
Oct 30 17:30:21.487749 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.486914    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3961d3d-0ada-44af-a530-2a7565e96257-kube-api-access-x989w\" (UniqueName: \"kubernetes.io/projected/c3961d3d-0ada-44af-a530-2a7565e96257-kube-api-access-x989w\") pod \"c3961d3d-0ada-44af-a530-2a7565e96257\" (UID: \"c3961d3d-0ada-44af-a530-2a7565e96257\") "
Oct 30 17:30:21.487749 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.486989    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b01cc039-2221-4716-8611-8d08004e4778-kube-api-access-nl4g9\" (UniqueName: \"kubernetes.io/projected/b01cc039-2221-4716-8611-8d08004e4778-kube-api-access-nl4g9\") pod \"b01cc039-2221-4716-8611-8d08004e4778\" (UID: \"b01cc039-2221-4716-8611-8d08004e4778\") "
Oct 30 17:30:21.487749 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.487214    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5668479f-566c-4d0a-9353-7a60f275673a-kube-api-access-n5sx2\" (UniqueName: \"kubernetes.io/projected/5668479f-566c-4d0a-9353-7a60f275673a-kube-api-access-n5sx2\") pod \"5668479f-566c-4d0a-9353-7a60f275673a\" (UID: \"5668479f-566c-4d0a-9353-7a60f275673a\") "
Oct 30 17:30:21.487749 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.487259    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5668479f-566c-4d0a-9353-7a60f275673a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5668479f-566c-4d0a-9353-7a60f275673a-maas-api-service-ca\") pod \"5668479f-566c-4d0a-9353-7a60f275673a\" (UID: \"5668479f-566c-4d0a-9353-7a60f275673a\") "
Oct 30 17:30:21.488221 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.488146    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l45sh\" (UniqueName: \"kubernetes.io/projected/f02fcdb6-55c4-4713-b726-ff1de3d3a7ba-kube-api-access-l45sh\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:30:21.488559 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.488531    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5668479f-566c-4d0a-9353-7a60f275673a-maas-api-service-ca" pod "5668479f-566c-4d0a-9353-7a60f275673a" (UID: "5668479f-566c-4d0a-9353-7a60f275673a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:30:21.489190 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.489170    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b01cc039-2221-4716-8611-8d08004e4778-maas-api-service-ca" pod "b01cc039-2221-4716-8611-8d08004e4778" (UID: "b01cc039-2221-4716-8611-8d08004e4778"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:30:21.489728 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.489708    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3961d3d-0ada-44af-a530-2a7565e96257-maas-api-service-ca" pod "c3961d3d-0ada-44af-a530-2a7565e96257" (UID: "c3961d3d-0ada-44af-a530-2a7565e96257"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:30:21.499444 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.499395    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b01cc039-2221-4716-8611-8d08004e4778-kube-api-access-nl4g9" pod "b01cc039-2221-4716-8611-8d08004e4778" (UID: "b01cc039-2221-4716-8611-8d08004e4778"). InnerVolumeSpecName "kube-api-access-nl4g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:30:21.499767 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.499749    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5668479f-566c-4d0a-9353-7a60f275673a-kube-api-access-n5sx2" pod "5668479f-566c-4d0a-9353-7a60f275673a" (UID: "5668479f-566c-4d0a-9353-7a60f275673a"). InnerVolumeSpecName "kube-api-access-n5sx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:30:21.505838 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.505810    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3961d3d-0ada-44af-a530-2a7565e96257-kube-api-access-x989w" pod "c3961d3d-0ada-44af-a530-2a7565e96257" (UID: "c3961d3d-0ada-44af-a530-2a7565e96257"). InnerVolumeSpecName "kube-api-access-x989w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:30:21.589297 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.589204    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b01cc039-2221-4716-8611-8d08004e4778-maas-api-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:30:21.589297 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.589232    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3961d3d-0ada-44af-a530-2a7565e96257-maas-api-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:30:21.589297 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.589242    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x989w\" (UniqueName: \"kubernetes.io/projected/c3961d3d-0ada-44af-a530-2a7565e96257-kube-api-access-x989w\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:30:21.589297 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.589251    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl4g9\" (UniqueName: \"kubernetes.io/projected/b01cc039-2221-4716-8611-8d08004e4778-kube-api-access-nl4g9\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:30:21.589297 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.589261    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5sx2\" (UniqueName: \"kubernetes.io/projected/5668479f-566c-4d0a-9353-7a60f275673a-kube-api-access-n5sx2\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:30:21.589297 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.589274    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5668479f-566c-4d0a-9353-7a60f275673a-maas-api-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:30:21.695863 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.695831    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12e09dc3-bfd6-4b8b-90ca-8fab73d0c922" path="/var/lib/kubelet/pods/12e09dc3-bfd6-4b8b-90ca-8fab73d0c922/volumes"
Oct 30 17:30:21.696325 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.696308    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35124e4a-306c-4d88-9321-d5de27812912" path="/var/lib/kubelet/pods/35124e4a-306c-4d88-9321-d5de27812912/volumes"
Oct 30 17:30:21.696779 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.696765    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d225e1b-96bc-4632-8088-60a96b84999c" path="/var/lib/kubelet/pods/5d225e1b-96bc-4632-8088-60a96b84999c/volumes"
Oct 30 17:30:21.697521 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.697503    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73952f5d-729f-4433-aeeb-21c31b18c1d9" path="/var/lib/kubelet/pods/73952f5d-729f-4433-aeeb-21c31b18c1d9/volumes"
Oct 30 17:30:21.697906 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.697893    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3" path="/var/lib/kubelet/pods/786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3/volumes"
Oct 30 17:30:21.698288 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.698276    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="993f683e-25cd-4f6e-beff-4da2e7c6ea69" path="/var/lib/kubelet/pods/993f683e-25cd-4f6e-beff-4da2e7c6ea69/volumes"
Oct 30 17:30:21.698945 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.698930    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9d1683a-8e8a-4717-90ad-a4d84861d2f1" path="/var/lib/kubelet/pods/c9d1683a-8e8a-4717-90ad-a4d84861d2f1/volumes"
Oct 30 17:30:21.699312 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:21.699300    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db5021f4-7307-4efa-8155-dc88fb2d95de" path="/var/lib/kubelet/pods/db5021f4-7307-4efa-8155-dc88fb2d95de/volumes"
Oct 30 17:30:22.049732 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:22.049685    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg" event={"ID":"f02fcdb6-55c4-4713-b726-ff1de3d3a7ba","Type":"ContainerDied","Data":"8914be209518b8e5500dd101839d42a62f1768a90d29018ff70d7b0bdfed1549"}
Oct 30 17:30:22.050206 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:22.049807    4433 scope.go:122] "RemoveContainer" containerID="f1b988bb7536457f43e09de6d1a06e77261f571b7c52d4fe1cdf65555a0ab929"
Oct 30 17:30:22.050206 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:22.049971    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg"
Oct 30 17:30:22.052882 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:22.052847    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws" event={"ID":"c3961d3d-0ada-44af-a530-2a7565e96257","Type":"ContainerDied","Data":"a3781df90c03fa2b80971c209460e1ab76a52557dcfce08664a21c4982ea6de5"}
Oct 30 17:30:22.053152 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:22.053134    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws"
Oct 30 17:30:22.056493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:22.056469    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl" event={"ID":"5668479f-566c-4d0a-9353-7a60f275673a","Type":"ContainerDied","Data":"02b70510d2168c30221c84f53d95bc08d4566f0e37e65612170f08c6e78a5bd8"}
Oct 30 17:30:22.056604 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:22.056570    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl"
Oct 30 17:30:22.058718 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:22.058693    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n"
Oct 30 17:30:22.058839 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:22.058708    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n" event={"ID":"b01cc039-2221-4716-8611-8d08004e4778","Type":"ContainerDied","Data":"22f88cfc3aab643c6560568f348afd071114c8dc2979e028737899dd74713533"}
Oct 30 17:30:22.074477 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:22.074454    4433 scope.go:122] "RemoveContainer" containerID="ff85e04bd71efd87bd1992ab15a4c5184c5bc001c775a8999cc1d7d96c69fcb0"
Oct 30 17:30:22.085148 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:22.085129    4433 scope.go:122] "RemoveContainer" containerID="827245adc9428d6c4ef9ae3b1f405f58883fd04e46d93d181f7c1fa745063c62"
Oct 30 17:30:22.094136 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:22.094112    4433 scope.go:122] "RemoveContainer" containerID="333b3ac01545205fa32a4cbf64f3fb9890c28048cafe9ca3af1df0aaa4670cb1"
Oct 30 17:30:22.104836 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:22.104811    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg"]
Oct 30 17:30:22.115697 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:22.115673    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-63e60d-29889690-kstqg"]
Oct 30 17:30:22.144012 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:22.143985    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws"]
Oct 30 17:30:22.148111 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:22.148088    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6d4f99-29889690-nlqws"]
Oct 30 17:30:22.180821 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:22.180787    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl"]
Oct 30 17:30:22.188699 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:22.188671    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b87f3c-29889690-242xl"]
Oct 30 17:30:22.225357 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:22.225329    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n"]
Oct 30 17:30:22.229514 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:22.229491    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4f406-29889690-mbj9n"]
Oct 30 17:30:23.697194 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:23.697160    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5668479f-566c-4d0a-9353-7a60f275673a" path="/var/lib/kubelet/pods/5668479f-566c-4d0a-9353-7a60f275673a/volumes"
Oct 30 17:30:23.697681 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:23.697665    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b01cc039-2221-4716-8611-8d08004e4778" path="/var/lib/kubelet/pods/b01cc039-2221-4716-8611-8d08004e4778/volumes"
Oct 30 17:30:23.698302 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:23.698283    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3961d3d-0ada-44af-a530-2a7565e96257" path="/var/lib/kubelet/pods/c3961d3d-0ada-44af-a530-2a7565e96257/volumes"
Oct 30 17:30:23.699040 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:23.699024    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f02fcdb6-55c4-4713-b726-ff1de3d3a7ba" path="/var/lib/kubelet/pods/f02fcdb6-55c4-4713-b726-ff1de3d3a7ba/volumes"
Oct 30 17:30:25.616970 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:25.616936    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:30:26.979099 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:26.979062    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:30:27.237234 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:27.237151    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq"
Oct 30 17:30:27.250021 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:27.249992    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq"
Oct 30 17:30:28.144297 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:28.144263    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq"
Oct 30 17:30:47.385679 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.385641    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"]
Oct 30 17:30:47.386564 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386543    4433 cpu_manager.go:414] "containerMap: removing container" podUID="12e09dc3-bfd6-4b8b-90ca-8fab73d0c922" containerName="cleanup"
Oct 30 17:30:47.386679 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386564    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12e09dc3-bfd6-4b8b-90ca-8fab73d0c922" containerName="cleanup"
Oct 30 17:30:47.386679 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386580    4433 cpu_manager.go:414] "containerMap: removing container" podUID="f02fcdb6-55c4-4713-b726-ff1de3d3a7ba" containerName="cleanup"
Oct 30 17:30:47.386679 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386588    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f02fcdb6-55c4-4713-b726-ff1de3d3a7ba" containerName="cleanup"
Oct 30 17:30:47.386679 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386599    4433 cpu_manager.go:414] "containerMap: removing container" podUID="b01cc039-2221-4716-8611-8d08004e4778" containerName="cleanup"
Oct 30 17:30:47.386679 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386607    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b01cc039-2221-4716-8611-8d08004e4778" containerName="cleanup"
Oct 30 17:30:47.386679 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386620    4433 cpu_manager.go:414] "containerMap: removing container" podUID="73952f5d-729f-4433-aeeb-21c31b18c1d9" containerName="cleanup"
Oct 30 17:30:47.386679 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386627    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73952f5d-729f-4433-aeeb-21c31b18c1d9" containerName="cleanup"
Oct 30 17:30:47.386679 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386643    4433 cpu_manager.go:414] "containerMap: removing container" podUID="786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3" containerName="cleanup"
Oct 30 17:30:47.386679 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386650    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3" containerName="cleanup"
Oct 30 17:30:47.386679 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386658    4433 cpu_manager.go:414] "containerMap: removing container" podUID="f02fcdb6-55c4-4713-b726-ff1de3d3a7ba" containerName="cleanup"
Oct 30 17:30:47.386679 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386665    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f02fcdb6-55c4-4713-b726-ff1de3d3a7ba" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386686    4433 cpu_manager.go:414] "containerMap: removing container" podUID="c3961d3d-0ada-44af-a530-2a7565e96257" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386693    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3961d3d-0ada-44af-a530-2a7565e96257" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386704    4433 cpu_manager.go:414] "containerMap: removing container" podUID="c3961d3d-0ada-44af-a530-2a7565e96257" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386711    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3961d3d-0ada-44af-a530-2a7565e96257" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386722    4433 cpu_manager.go:414] "containerMap: removing container" podUID="c9d1683a-8e8a-4717-90ad-a4d84861d2f1" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386729    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9d1683a-8e8a-4717-90ad-a4d84861d2f1" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386745    4433 cpu_manager.go:414] "containerMap: removing container" podUID="35124e4a-306c-4d88-9321-d5de27812912" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386752    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35124e4a-306c-4d88-9321-d5de27812912" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386764    4433 cpu_manager.go:414] "containerMap: removing container" podUID="5d225e1b-96bc-4632-8088-60a96b84999c" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386771    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d225e1b-96bc-4632-8088-60a96b84999c" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386781    4433 cpu_manager.go:414] "containerMap: removing container" podUID="5668479f-566c-4d0a-9353-7a60f275673a" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386788    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5668479f-566c-4d0a-9353-7a60f275673a" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386803    4433 cpu_manager.go:414] "containerMap: removing container" podUID="12e09dc3-bfd6-4b8b-90ca-8fab73d0c922" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386810    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12e09dc3-bfd6-4b8b-90ca-8fab73d0c922" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386819    4433 cpu_manager.go:414] "containerMap: removing container" podUID="35124e4a-306c-4d88-9321-d5de27812912" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386826    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35124e4a-306c-4d88-9321-d5de27812912" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386836    4433 cpu_manager.go:414] "containerMap: removing container" podUID="db5021f4-7307-4efa-8155-dc88fb2d95de" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386843    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db5021f4-7307-4efa-8155-dc88fb2d95de" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386852    4433 cpu_manager.go:414] "containerMap: removing container" podUID="5d225e1b-96bc-4632-8088-60a96b84999c" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386861    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d225e1b-96bc-4632-8088-60a96b84999c" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386875    4433 cpu_manager.go:414] "containerMap: removing container" podUID="5d225e1b-96bc-4632-8088-60a96b84999c" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386882    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d225e1b-96bc-4632-8088-60a96b84999c" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386893    4433 cpu_manager.go:414] "containerMap: removing container" podUID="993f683e-25cd-4f6e-beff-4da2e7c6ea69" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386899    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="993f683e-25cd-4f6e-beff-4da2e7c6ea69" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386909    4433 cpu_manager.go:414] "containerMap: removing container" podUID="c3961d3d-0ada-44af-a530-2a7565e96257" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386916    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3961d3d-0ada-44af-a530-2a7565e96257" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386938    4433 cpu_manager.go:414] "containerMap: removing container" podUID="db5021f4-7307-4efa-8155-dc88fb2d95de" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386945    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db5021f4-7307-4efa-8155-dc88fb2d95de" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386955    4433 cpu_manager.go:414] "containerMap: removing container" podUID="5668479f-566c-4d0a-9353-7a60f275673a" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386962    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5668479f-566c-4d0a-9353-7a60f275673a" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386974    4433 cpu_manager.go:414] "containerMap: removing container" podUID="c9d1683a-8e8a-4717-90ad-a4d84861d2f1" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386982    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9d1683a-8e8a-4717-90ad-a4d84861d2f1" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.386996    4433 cpu_manager.go:414] "containerMap: removing container" podUID="993f683e-25cd-4f6e-beff-4da2e7c6ea69" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387004    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="993f683e-25cd-4f6e-beff-4da2e7c6ea69" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387016    4433 cpu_manager.go:414] "containerMap: removing container" podUID="b01cc039-2221-4716-8611-8d08004e4778" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387023    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b01cc039-2221-4716-8611-8d08004e4778" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387042    4433 cpu_manager.go:414] "containerMap: removing container" podUID="786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387049    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387062    4433 cpu_manager.go:414] "containerMap: removing container" podUID="993f683e-25cd-4f6e-beff-4da2e7c6ea69" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387069    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="993f683e-25cd-4f6e-beff-4da2e7c6ea69" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387085    4433 cpu_manager.go:414] "containerMap: removing container" podUID="35124e4a-306c-4d88-9321-d5de27812912" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387092    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35124e4a-306c-4d88-9321-d5de27812912" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387106    4433 cpu_manager.go:414] "containerMap: removing container" podUID="73952f5d-729f-4433-aeeb-21c31b18c1d9" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387113    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73952f5d-729f-4433-aeeb-21c31b18c1d9" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387135    4433 cpu_manager.go:414] "containerMap: removing container" podUID="db5021f4-7307-4efa-8155-dc88fb2d95de" containerName="cleanup"
Oct 30 17:30:47.387258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387143    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db5021f4-7307-4efa-8155-dc88fb2d95de" containerName="cleanup"
Oct 30 17:30:47.389561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387304    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="12e09dc3-bfd6-4b8b-90ca-8fab73d0c922" containerName="cleanup"
Oct 30 17:30:47.389561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387317    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="35124e4a-306c-4d88-9321-d5de27812912" containerName="cleanup"
Oct 30 17:30:47.389561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387327    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="db5021f4-7307-4efa-8155-dc88fb2d95de" containerName="cleanup"
Oct 30 17:30:47.389561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387337    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d225e1b-96bc-4632-8088-60a96b84999c" containerName="cleanup"
Oct 30 17:30:47.389561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387346    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d225e1b-96bc-4632-8088-60a96b84999c" containerName="cleanup"
Oct 30 17:30:47.389561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387356    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="993f683e-25cd-4f6e-beff-4da2e7c6ea69" containerName="cleanup"
Oct 30 17:30:47.389561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387367    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3961d3d-0ada-44af-a530-2a7565e96257" containerName="cleanup"
Oct 30 17:30:47.389561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387375    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="b01cc039-2221-4716-8611-8d08004e4778" containerName="cleanup"
Oct 30 17:30:47.389561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387384    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="db5021f4-7307-4efa-8155-dc88fb2d95de" containerName="cleanup"
Oct 30 17:30:47.389561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387392    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="5668479f-566c-4d0a-9353-7a60f275673a" containerName="cleanup"
Oct 30 17:30:47.389561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387417    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9d1683a-8e8a-4717-90ad-a4d84861d2f1" containerName="cleanup"
Oct 30 17:30:47.389561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387425    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9d1683a-8e8a-4717-90ad-a4d84861d2f1" containerName="cleanup"
Oct 30 17:30:47.389561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387435    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="993f683e-25cd-4f6e-beff-4da2e7c6ea69" containerName="cleanup"
Oct 30 17:30:47.389561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387445    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="b01cc039-2221-4716-8611-8d08004e4778" containerName="cleanup"
Oct 30 17:30:47.389561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387455    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3" containerName="cleanup"
Oct 30 17:30:47.389561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387463    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="993f683e-25cd-4f6e-beff-4da2e7c6ea69" containerName="cleanup"
Oct 30 17:30:47.389561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387472    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="35124e4a-306c-4d88-9321-d5de27812912" containerName="cleanup"
Oct 30 17:30:47.389561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387479    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="73952f5d-729f-4433-aeeb-21c31b18c1d9" containerName="cleanup"
Oct 30 17:30:47.389561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387491    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="73952f5d-729f-4433-aeeb-21c31b18c1d9" containerName="cleanup"
Oct 30 17:30:47.389561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387499    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3" containerName="cleanup"
Oct 30 17:30:47.389561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387508    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="f02fcdb6-55c4-4713-b726-ff1de3d3a7ba" containerName="cleanup"
Oct 30 17:30:47.389561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387516    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="5668479f-566c-4d0a-9353-7a60f275673a" containerName="cleanup"
Oct 30 17:30:47.389561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387527    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="12e09dc3-bfd6-4b8b-90ca-8fab73d0c922" containerName="cleanup"
Oct 30 17:30:47.389561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387537    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="f02fcdb6-55c4-4713-b726-ff1de3d3a7ba" containerName="cleanup"
Oct 30 17:30:47.389561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387547    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="73952f5d-729f-4433-aeeb-21c31b18c1d9" containerName="cleanup"
Oct 30 17:30:47.389561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387557    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3" containerName="cleanup"
Oct 30 17:30:47.389561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387566    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="f02fcdb6-55c4-4713-b726-ff1de3d3a7ba" containerName="cleanup"
Oct 30 17:30:47.389561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387578    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3961d3d-0ada-44af-a530-2a7565e96257" containerName="cleanup"
Oct 30 17:30:47.389561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.387590    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3961d3d-0ada-44af-a530-2a7565e96257" containerName="cleanup"
Oct 30 17:30:47.393353 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.393325    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:30:47.416268 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.416222    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-9e9ce7-openshift-default-dockercfg-5tgtf\""
Oct 30 17:30:47.437879 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.437838    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"]
Oct 30 17:30:47.558430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.552629    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34119e6e-443a-40df-a54a-0f03b4e34553-workload-socket\") pod \"e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") " pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:30:47.558430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.552832    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34119e6e-443a-40df-a54a-0f03b4e34553-istio-data\") pod \"e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") " pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:30:47.558430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.552966    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t7kz\" (UniqueName: \"kubernetes.io/projected/34119e6e-443a-40df-a54a-0f03b4e34553-kube-api-access-8t7kz\") pod \"e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") " pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:30:47.558430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.553018    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34119e6e-443a-40df-a54a-0f03b4e34553-istio-podinfo\") pod \"e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") " pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:30:47.558430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.553097    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34119e6e-443a-40df-a54a-0f03b4e34553-credential-socket\") pod \"e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") " pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:30:47.558430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.553127    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34119e6e-443a-40df-a54a-0f03b4e34553-istio-envoy\") pod \"e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") " pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:30:47.558430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.553183    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34119e6e-443a-40df-a54a-0f03b4e34553-workload-certs\") pod \"e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") " pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:30:47.558430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.553213    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34119e6e-443a-40df-a54a-0f03b4e34553-istio-token\") pod \"e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") " pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:30:47.558430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.553360    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34119e6e-443a-40df-a54a-0f03b4e34553-istiod-ca-cert\") pod \"e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") " pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:30:47.661002 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.660358    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t7kz\" (UniqueName: \"kubernetes.io/projected/34119e6e-443a-40df-a54a-0f03b4e34553-kube-api-access-8t7kz\") pod \"e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") " pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:30:47.661002 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.660443    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34119e6e-443a-40df-a54a-0f03b4e34553-istio-podinfo\") pod \"e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") " pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:30:47.661002 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.660509    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34119e6e-443a-40df-a54a-0f03b4e34553-credential-socket\") pod \"e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") " pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:30:47.661002 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.660536    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34119e6e-443a-40df-a54a-0f03b4e34553-istio-envoy\") pod \"e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") " pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:30:47.661002 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.660582    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34119e6e-443a-40df-a54a-0f03b4e34553-workload-certs\") pod \"e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") " pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:30:47.661002 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.660607    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34119e6e-443a-40df-a54a-0f03b4e34553-istio-token\") pod \"e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") " pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:30:47.661002 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.660721    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34119e6e-443a-40df-a54a-0f03b4e34553-istiod-ca-cert\") pod \"e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") " pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:30:47.661002 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.660845    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34119e6e-443a-40df-a54a-0f03b4e34553-workload-socket\") pod \"e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") " pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:30:47.661002 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.660926    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34119e6e-443a-40df-a54a-0f03b4e34553-istio-data\") pod \"e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") " pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:30:47.661643 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.661454    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34119e6e-443a-40df-a54a-0f03b4e34553-istio-data\") pod \"e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") " pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:30:47.667424 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.662947    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34119e6e-443a-40df-a54a-0f03b4e34553-credential-socket\") pod \"e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") " pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:30:47.667424 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.663617    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34119e6e-443a-40df-a54a-0f03b4e34553-istiod-ca-cert\") pod \"e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") " pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:30:47.667424 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.664153    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34119e6e-443a-40df-a54a-0f03b4e34553-workload-socket\") pod \"e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") " pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:30:47.667706 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.667453    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34119e6e-443a-40df-a54a-0f03b4e34553-workload-certs\") pod \"e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") " pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:30:47.687433 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.685042    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34119e6e-443a-40df-a54a-0f03b4e34553-istio-podinfo\") pod \"e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") " pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:30:47.711014 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.710752    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34119e6e-443a-40df-a54a-0f03b4e34553-istio-envoy\") pod \"e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") " pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:30:47.723490 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.723459    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t7kz\" (UniqueName: \"kubernetes.io/projected/34119e6e-443a-40df-a54a-0f03b4e34553-kube-api-access-8t7kz\") pod \"e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") " pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:30:47.735984 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:47.735119    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34119e6e-443a-40df-a54a-0f03b4e34553-istio-token\") pod \"e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") " pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:30:48.013426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:48.013179    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:30:48.366574 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:48.366534    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"]
Oct 30 17:30:48.368022 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:30:48.367993    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34119e6e_443a_40df_a54a_0f03b4e34553.slice/crio-56db25acc3ea20484cd332afdacc5ccb855f1c7c3ee4a4ad52844ef9e52271f3 WatchSource:0}: Error finding container 56db25acc3ea20484cd332afdacc5ccb855f1c7c3ee4a4ad52844ef9e52271f3: Status 404 returned error can't find the container with id 56db25acc3ea20484cd332afdacc5ccb855f1c7c3ee4a4ad52844ef9e52271f3
Oct 30 17:30:48.371266 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:48.371230    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:30:48.371346 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:48.371304    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:30:48.371346 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:48.371331    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:30:49.302131 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:49.302084    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg" event={"ID":"34119e6e-443a-40df-a54a-0f03b4e34553","Type":"ContainerStarted","Data":"db7028eeec58108cdd61659d39cc8be05b3e56e49b004f2268765a17af8eb4c2"}
Oct 30 17:30:49.302625 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:49.302153    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg" event={"ID":"34119e6e-443a-40df-a54a-0f03b4e34553","Type":"ContainerStarted","Data":"56db25acc3ea20484cd332afdacc5ccb855f1c7c3ee4a4ad52844ef9e52271f3"}
Oct 30 17:30:50.014249 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:50.014207    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:30:50.022296 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:50.022266    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:30:50.092077 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:50.092012    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg" podStartSLOduration=3.091991023 podStartE2EDuration="3.091991023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:49.395487499 +0000 UTC m=+2625.830389706" watchObservedRunningTime="2026-10-30 17:30:50.091991023 +0000 UTC m=+2626.526893231"
Oct 30 17:30:50.310107 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:50.310020    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:30:50.311232 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:50.311207    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:30:54.010438 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:54.010321    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-84c5bf86f8-54rcv"]
Oct 30 17:30:54.132443 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:54.125305    4433 cpu_manager.go:414] "containerMap: removing container" podUID="5668479f-566c-4d0a-9353-7a60f275673a" containerName="cleanup"
Oct 30 17:30:54.132443 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:54.125342    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5668479f-566c-4d0a-9353-7a60f275673a" containerName="cleanup"
Oct 30 17:30:54.132443 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:54.125374    4433 cpu_manager.go:414] "containerMap: removing container" podUID="786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3" containerName="cleanup"
Oct 30 17:30:54.132443 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:54.125383    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="786ee82f-a3a6-4a48-b5a6-c98fdf2a45b3" containerName="cleanup"
Oct 30 17:30:54.132443 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:54.125455    4433 cpu_manager.go:414] "containerMap: removing container" podUID="b01cc039-2221-4716-8611-8d08004e4778" containerName="cleanup"
Oct 30 17:30:54.132443 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:54.125464    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b01cc039-2221-4716-8611-8d08004e4778" containerName="cleanup"
Oct 30 17:30:54.132443 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:54.125479    4433 cpu_manager.go:414] "containerMap: removing container" podUID="73952f5d-729f-4433-aeeb-21c31b18c1d9" containerName="cleanup"
Oct 30 17:30:54.132443 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:54.125486    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73952f5d-729f-4433-aeeb-21c31b18c1d9" containerName="cleanup"
Oct 30 17:30:54.132443 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:54.125502    4433 cpu_manager.go:414] "containerMap: removing container" podUID="f02fcdb6-55c4-4713-b726-ff1de3d3a7ba" containerName="cleanup"
Oct 30 17:30:54.132443 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:54.125509    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f02fcdb6-55c4-4713-b726-ff1de3d3a7ba" containerName="cleanup"
Oct 30 17:30:54.132443 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:54.125528    4433 cpu_manager.go:414] "containerMap: removing container" podUID="c9d1683a-8e8a-4717-90ad-a4d84861d2f1" containerName="cleanup"
Oct 30 17:30:54.132443 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:54.125535    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9d1683a-8e8a-4717-90ad-a4d84861d2f1" containerName="cleanup"
Oct 30 17:30:54.132443 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:54.125555    4433 cpu_manager.go:414] "containerMap: removing container" podUID="12e09dc3-bfd6-4b8b-90ca-8fab73d0c922" containerName="cleanup"
Oct 30 17:30:54.132443 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:54.125563    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12e09dc3-bfd6-4b8b-90ca-8fab73d0c922" containerName="cleanup"
Oct 30 17:30:54.132443 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:54.125743    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="12e09dc3-bfd6-4b8b-90ca-8fab73d0c922" containerName="cleanup"
Oct 30 17:30:54.132443 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:54.125757    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="35124e4a-306c-4d88-9321-d5de27812912" containerName="cleanup"
Oct 30 17:30:54.132443 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:54.125770    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d225e1b-96bc-4632-8088-60a96b84999c" containerName="cleanup"
Oct 30 17:30:54.132443 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:54.125781    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="db5021f4-7307-4efa-8155-dc88fb2d95de" containerName="cleanup"
Oct 30 17:30:54.132443 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:54.125793    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9d1683a-8e8a-4717-90ad-a4d84861d2f1" containerName="cleanup"
Oct 30 17:30:54.132443 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:54.125801    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="b01cc039-2221-4716-8611-8d08004e4778" containerName="cleanup"
Oct 30 17:30:54.132443 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:54.125816    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="5668479f-566c-4d0a-9353-7a60f275673a" containerName="cleanup"
Oct 30 17:30:54.232450 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:54.231617    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-84c5bf86f8-54rcv"]
Oct 30 17:30:54.232450 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:54.231745    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-84c5bf86f8-54rcv"
Oct 30 17:30:54.263189 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:54.263095    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9e9ce7\""
Oct 30 17:30:54.263311 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:54.263218    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-9e9ce7-dockercfg-rllrl\""
Oct 30 17:30:54.403260 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:54.403214    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68747a7e-891d-465e-9682-df5b696b50ff-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9e9ce7-84c5bf86f8-54rcv\" (UID: \"68747a7e-891d-465e-9682-df5b696b50ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-84c5bf86f8-54rcv"
Oct 30 17:30:54.403476 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:54.403375    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94bvk\" (UniqueName: \"kubernetes.io/projected/68747a7e-891d-465e-9682-df5b696b50ff-kube-api-access-94bvk\") pod \"payload-processing-e2e-ipp-b-9e9ce7-84c5bf86f8-54rcv\" (UID: \"68747a7e-891d-465e-9682-df5b696b50ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-84c5bf86f8-54rcv"
Oct 30 17:30:54.505164 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:54.505127    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68747a7e-891d-465e-9682-df5b696b50ff-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9e9ce7-84c5bf86f8-54rcv\" (UID: \"68747a7e-891d-465e-9682-df5b696b50ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-84c5bf86f8-54rcv"
Oct 30 17:30:54.505383 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:54.505227    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94bvk\" (UniqueName: \"kubernetes.io/projected/68747a7e-891d-465e-9682-df5b696b50ff-kube-api-access-94bvk\") pod \"payload-processing-e2e-ipp-b-9e9ce7-84c5bf86f8-54rcv\" (UID: \"68747a7e-891d-465e-9682-df5b696b50ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-84c5bf86f8-54rcv"
Oct 30 17:30:54.505955 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:54.505928    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68747a7e-891d-465e-9682-df5b696b50ff-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9e9ce7-84c5bf86f8-54rcv\" (UID: \"68747a7e-891d-465e-9682-df5b696b50ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-84c5bf86f8-54rcv"
Oct 30 17:30:54.533303 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:54.533226    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94bvk\" (UniqueName: \"kubernetes.io/projected/68747a7e-891d-465e-9682-df5b696b50ff-kube-api-access-94bvk\") pod \"payload-processing-e2e-ipp-b-9e9ce7-84c5bf86f8-54rcv\" (UID: \"68747a7e-891d-465e-9682-df5b696b50ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-84c5bf86f8-54rcv"
Oct 30 17:30:54.569034 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:54.568996    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-84c5bf86f8-54rcv"
Oct 30 17:30:54.913969 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:54.913937    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-84c5bf86f8-54rcv"]
Oct 30 17:30:54.915819 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:30:54.915794    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68747a7e_891d_465e_9682_df5b696b50ff.slice/crio-97495338e8b0816b78cd4aa149431152164691a923caf78b5f62e791a190c18c WatchSource:0}: Error finding container 97495338e8b0816b78cd4aa149431152164691a923caf78b5f62e791a190c18c: Status 404 returned error can't find the container with id 97495338e8b0816b78cd4aa149431152164691a923caf78b5f62e791a190c18c
Oct 30 17:30:55.355637 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:55.355597    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-84c5bf86f8-54rcv" event={"ID":"68747a7e-891d-465e-9682-df5b696b50ff","Type":"ContainerStarted","Data":"b1db6655054b803b730f18f43637ca32b68a5e0fa66dba8d43f00b388d2bf26c"}
Oct 30 17:30:55.356052 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:55.355665    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-84c5bf86f8-54rcv"
Oct 30 17:30:55.356052 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:55.355685    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-84c5bf86f8-54rcv" event={"ID":"68747a7e-891d-465e-9682-df5b696b50ff","Type":"ContainerStarted","Data":"97495338e8b0816b78cd4aa149431152164691a923caf78b5f62e791a190c18c"}
Oct 30 17:30:55.408520 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:30:55.408456    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-84c5bf86f8-54rcv" podStartSLOduration=2.408440354 podStartE2EDuration="2.408440354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:55.39534555 +0000 UTC m=+2631.830247764" watchObservedRunningTime="2026-10-30 17:30:55.408440354 +0000 UTC m=+2631.843342551"
Oct 30 17:30:56.981251 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:30:56.981205    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:31:06.364975 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:06.364938    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-84c5bf86f8-54rcv"
Oct 30 17:31:17.233488 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:17.233390    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-5b45f9975-lrcs9"]
Oct 30 17:31:22.234193 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:22.234113    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-5b45f9975-lrcs9" podUID="3938c343-9631-413d-839a-76c566c23d3d" containerName="payload-pre-processing" containerID="cri-o://9f58ad4abd7f1307f18c0b6615fc0fb5772f8cdecdc3ce6129223f76cc45ece9" gracePeriod=25
Oct 30 17:31:22.535792 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:22.535765    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-5b45f9975-lrcs9"
Oct 30 17:31:22.594782 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:22.594743    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3938c343-9631-413d-839a-76c566c23d3d-kube-api-access-8rk66\" (UniqueName: \"kubernetes.io/projected/3938c343-9631-413d-839a-76c566c23d3d-kube-api-access-8rk66\") pod \"3938c343-9631-413d-839a-76c566c23d3d\" (UID: \"3938c343-9631-413d-839a-76c566c23d3d\") "
Oct 30 17:31:22.594984 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:22.594802    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3938c343-9631-413d-839a-76c566c23d3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3938c343-9631-413d-839a-76c566c23d3d-plugins-config-volume\") pod \"3938c343-9631-413d-839a-76c566c23d3d\" (UID: \"3938c343-9631-413d-839a-76c566c23d3d\") "
Oct 30 17:31:22.596467 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:22.596106    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3938c343-9631-413d-839a-76c566c23d3d-plugins-config-volume" pod "3938c343-9631-413d-839a-76c566c23d3d" (UID: "3938c343-9631-413d-839a-76c566c23d3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:31:22.600469 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:22.599351    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3938c343-9631-413d-839a-76c566c23d3d-kube-api-access-8rk66" pod "3938c343-9631-413d-839a-76c566c23d3d" (UID: "3938c343-9631-413d-839a-76c566c23d3d"). InnerVolumeSpecName "kube-api-access-8rk66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:31:22.608123 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:22.608086    4433 generic.go:356] "Generic (PLEG): container finished" podID="3938c343-9631-413d-839a-76c566c23d3d" containerID="9f58ad4abd7f1307f18c0b6615fc0fb5772f8cdecdc3ce6129223f76cc45ece9" exitCode=0
Oct 30 17:31:22.608294 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:22.608134    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-5b45f9975-lrcs9"
Oct 30 17:31:22.608294 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:22.608153    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-5b45f9975-lrcs9" event={"ID":"3938c343-9631-413d-839a-76c566c23d3d","Type":"ContainerDied","Data":"9f58ad4abd7f1307f18c0b6615fc0fb5772f8cdecdc3ce6129223f76cc45ece9"}
Oct 30 17:31:22.608294 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:22.608233    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-5b45f9975-lrcs9" event={"ID":"3938c343-9631-413d-839a-76c566c23d3d","Type":"ContainerDied","Data":"65cd8fe468df8a79d117b1e0cbf7bb09cf1c5aa62f978f9f5db7c156105c7ecb"}
Oct 30 17:31:22.608294 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:22.608267    4433 scope.go:122] "RemoveContainer" containerID="9f58ad4abd7f1307f18c0b6615fc0fb5772f8cdecdc3ce6129223f76cc45ece9"
Oct 30 17:31:22.633280 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:22.633254    4433 scope.go:122] "RemoveContainer" containerID="9f58ad4abd7f1307f18c0b6615fc0fb5772f8cdecdc3ce6129223f76cc45ece9"
Oct 30 17:31:22.634981 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:31:22.634940    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f58ad4abd7f1307f18c0b6615fc0fb5772f8cdecdc3ce6129223f76cc45ece9\": container with ID starting with 9f58ad4abd7f1307f18c0b6615fc0fb5772f8cdecdc3ce6129223f76cc45ece9 not found: ID does not exist" containerID="9f58ad4abd7f1307f18c0b6615fc0fb5772f8cdecdc3ce6129223f76cc45ece9"
Oct 30 17:31:22.635125 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:22.634992    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f58ad4abd7f1307f18c0b6615fc0fb5772f8cdecdc3ce6129223f76cc45ece9"} err="failed to get container status \"9f58ad4abd7f1307f18c0b6615fc0fb5772f8cdecdc3ce6129223f76cc45ece9\": rpc error: code = NotFound desc = could not find container \"9f58ad4abd7f1307f18c0b6615fc0fb5772f8cdecdc3ce6129223f76cc45ece9\": container with ID starting with 9f58ad4abd7f1307f18c0b6615fc0fb5772f8cdecdc3ce6129223f76cc45ece9 not found: ID does not exist"
Oct 30 17:31:22.659937 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:22.659899    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-5b45f9975-lrcs9"]
Oct 30 17:31:22.663963 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:22.663930    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-93e020-5b45f9975-lrcs9"]
Oct 30 17:31:22.697138 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:22.697098    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rk66\" (UniqueName: \"kubernetes.io/projected/3938c343-9631-413d-839a-76c566c23d3d-kube-api-access-8rk66\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:31:22.697138 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:22.697134    4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3938c343-9631-413d-839a-76c566c23d3d-plugins-config-volume\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:31:23.700249 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:23.700213    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3938c343-9631-413d-839a-76c566c23d3d" path="/var/lib/kubelet/pods/3938c343-9631-413d-839a-76c566c23d3d/volumes"
Oct 30 17:31:26.978611 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:31:26.978572    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:31:28.538798 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:28.538763    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"]
Oct 30 17:31:28.539455 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:28.539438    4433 cpu_manager.go:414] "containerMap: removing container" podUID="3938c343-9631-413d-839a-76c566c23d3d" containerName="payload-pre-processing"
Oct 30 17:31:28.539509 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:28.539456    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3938c343-9631-413d-839a-76c566c23d3d" containerName="payload-pre-processing"
Oct 30 17:31:28.539581 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:28.539571    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="3938c343-9631-413d-839a-76c566c23d3d" containerName="payload-pre-processing"
Oct 30 17:31:28.544581 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:28.544556    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:31:28.576568 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:28.576526    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"]
Oct 30 17:31:28.582566 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:28.582531    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-4fb821-openshift-default-dockercfg-xqzrx\""
Oct 30 17:31:28.698813 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:28.698767    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-istio-podinfo\") pod \"e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") " pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:31:28.698980 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:28.698876    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-credential-socket\") pod \"e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") " pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:31:28.699026 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:28.698986    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-workload-certs\") pod \"e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") " pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:31:28.699116 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:28.699063    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-istio-data\") pod \"e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") " pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:31:28.699116 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:28.699107    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-istio-envoy\") pod \"e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") " pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:31:28.699194 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:28.699136    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsn5w\" (UniqueName: \"kubernetes.io/projected/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-kube-api-access-qsn5w\") pod \"e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") " pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:31:28.699258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:28.699236    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-istio-token\") pod \"e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") " pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:31:28.699335 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:28.699321    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-workload-socket\") pod \"e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") " pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:31:28.699436 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:28.699418    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-istiod-ca-cert\") pod \"e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") " pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:31:28.808478 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:28.807375    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-istio-podinfo\") pod \"e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") " pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:31:28.808478 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:28.807475    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-credential-socket\") pod \"e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") " pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:31:28.808478 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:28.807568    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-workload-certs\") pod \"e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") " pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:31:28.808478 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:28.807657    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-istio-data\") pod \"e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") " pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:31:28.808478 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:28.807701    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-istio-envoy\") pod \"e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") " pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:31:28.808478 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:28.807732    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsn5w\" (UniqueName: \"kubernetes.io/projected/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-kube-api-access-qsn5w\") pod \"e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") " pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:31:28.808478 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:28.807818    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-istio-token\") pod \"e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") " pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:31:28.808478 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:28.807936    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-workload-socket\") pod \"e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") " pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:31:28.808478 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:28.807974    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-istiod-ca-cert\") pod \"e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") " pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:31:28.809118 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:28.808941    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-istiod-ca-cert\") pod \"e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") " pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:31:28.844423 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:28.811989    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-workload-certs\") pod \"e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") " pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:31:28.844423 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:28.812237    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-credential-socket\") pod \"e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") " pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:31:28.844423 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:28.812840    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-istio-data\") pod \"e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") " pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:31:28.844423 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:28.813262    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-workload-socket\") pod \"e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") " pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:31:28.844423 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:28.823383    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-istio-podinfo\") pod \"e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") " pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:31:28.844423 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:28.840551    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-istio-envoy\") pod \"e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") " pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:31:28.846053 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:28.846023    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-istio-token\") pod \"e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") " pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:31:28.870438 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:28.869375    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsn5w\" (UniqueName: \"kubernetes.io/projected/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-kube-api-access-qsn5w\") pod \"e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") " pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:31:28.877804 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:28.877773    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:31:29.345077 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:29.345035    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"]
Oct 30 17:31:29.347472 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:31:29.347443    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fcae069_8f9f_47e5_8d3a_82c248cf3c31.slice/crio-bb5779c2eb0baa5c0cd010863d9dea828667f1bf9d5338cb4f13e8edee9e6533 WatchSource:0}: Error finding container bb5779c2eb0baa5c0cd010863d9dea828667f1bf9d5338cb4f13e8edee9e6533: Status 404 returned error can't find the container with id bb5779c2eb0baa5c0cd010863d9dea828667f1bf9d5338cb4f13e8edee9e6533
Oct 30 17:31:29.351230 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:29.351191    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:31:29.351358 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:29.351285    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:31:29.351358 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:29.351319    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:31:29.680661 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:29.680615    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft" event={"ID":"5fcae069-8f9f-47e5-8d3a-82c248cf3c31","Type":"ContainerStarted","Data":"0468bdc637faecdad7c1817dd916908c2917732fc4d55b05b4779401cb8f5afc"}
Oct 30 17:31:29.681117 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:29.680726    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft" event={"ID":"5fcae069-8f9f-47e5-8d3a-82c248cf3c31","Type":"ContainerStarted","Data":"bb5779c2eb0baa5c0cd010863d9dea828667f1bf9d5338cb4f13e8edee9e6533"}
Oct 30 17:31:29.749921 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:29.749856    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft" podStartSLOduration=1.749840792 podStartE2EDuration="1.749840792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:31:29.73696107 +0000 UTC m=+2666.171863291" watchObservedRunningTime="2026-10-30 17:31:29.749840792 +0000 UTC m=+2666.184742998"
Oct 30 17:31:29.879024 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:29.878988    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:31:30.885965 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:30.885935    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:31:31.697444 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:31.697374    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:31:31.697616 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:31.697529    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:31:38.427760 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:38.427722    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk"]
Oct 30 17:31:43.813648 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:43.813609    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk"]
Oct 30 17:31:43.814079 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:43.813860    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk" podUID="8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545" containerName="maas-api" containerID="cri-o://f151bad01a5375492112caaedb43662cfdffe673c14d62560eee30c9b4083341" gracePeriod=30
Oct 30 17:31:44.214427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:44.199611    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk"
Oct 30 17:31:44.322426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:44.321796    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545-maas-api-tls\") pod \"8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545\" (UID: \"8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545\") "
Oct 30 17:31:44.322426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:44.321937    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545-kube-api-access-l9z8s\" (UniqueName: \"kubernetes.io/projected/8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545-kube-api-access-l9z8s\") pod \"8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545\" (UID: \"8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545\") "
Oct 30 17:31:44.373485 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:44.348115    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545-metrics-certs\") pod \"8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545\" (UID: \"8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545\") "
Oct 30 17:31:44.373485 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:44.348339    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545-openshift-service-ca\") pod \"8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545\" (UID: \"8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545\") "
Oct 30 17:31:44.373485 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:44.352763    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545-openshift-service-ca" pod "8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545" (UID: "8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:31:44.373485 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:44.372336    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545-metrics-certs" pod "8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545" (UID: "8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:31:44.381523 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:44.377524    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545-maas-api-tls" pod "8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545" (UID: "8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:31:44.381523 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:44.377841    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545-kube-api-access-l9z8s" pod "8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545" (UID: "8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545"). InnerVolumeSpecName "kube-api-access-l9z8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:31:44.454716 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:44.454680    4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545-metrics-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:31:44.454716 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:44.454717    4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545-openshift-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:31:44.454977 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:44.454732    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545-maas-api-tls\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:31:44.454977 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:44.454745    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9z8s\" (UniqueName: \"kubernetes.io/projected/8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545-kube-api-access-l9z8s\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:31:44.888473 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:44.888432    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk" event={"ID":"8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545","Type":"ContainerDied","Data":"f151bad01a5375492112caaedb43662cfdffe673c14d62560eee30c9b4083341"}
Oct 30 17:31:44.888968 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:44.888478    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk"
Oct 30 17:31:44.888968 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:44.888512    4433 scope.go:122] "RemoveContainer" containerID="f151bad01a5375492112caaedb43662cfdffe673c14d62560eee30c9b4083341"
Oct 30 17:31:44.888968 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:44.888442    4433 generic.go:356] "Generic (PLEG): container finished" podID="8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545" containerID="f151bad01a5375492112caaedb43662cfdffe673c14d62560eee30c9b4083341" exitCode=0
Oct 30 17:31:44.888968 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:44.888616    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk" event={"ID":"8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545","Type":"ContainerDied","Data":"665befa699347cfeaced234853ed9dc9f00de651cfeaf166c1565dc02ed44d16"}
Oct 30 17:31:44.900476 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:44.900396    4433 scope.go:122] "RemoveContainer" containerID="f151bad01a5375492112caaedb43662cfdffe673c14d62560eee30c9b4083341"
Oct 30 17:31:44.900794 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:31:44.900774    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f151bad01a5375492112caaedb43662cfdffe673c14d62560eee30c9b4083341\": container with ID starting with f151bad01a5375492112caaedb43662cfdffe673c14d62560eee30c9b4083341 not found: ID does not exist" containerID="f151bad01a5375492112caaedb43662cfdffe673c14d62560eee30c9b4083341"
Oct 30 17:31:44.900907 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:44.900799    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f151bad01a5375492112caaedb43662cfdffe673c14d62560eee30c9b4083341"} err="failed to get container status \"f151bad01a5375492112caaedb43662cfdffe673c14d62560eee30c9b4083341\": rpc error: code = NotFound desc = could not find container \"f151bad01a5375492112caaedb43662cfdffe673c14d62560eee30c9b4083341\": container with ID starting with f151bad01a5375492112caaedb43662cfdffe673c14d62560eee30c9b4083341 not found: ID does not exist"
Oct 30 17:31:44.940753 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:44.940723    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk"]
Oct 30 17:31:44.948021 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:31:44.947986    4433 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd302f4_10c7_4e0b_a5f5_fd3ea26d2545.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd302f4_10c7_4e0b_a5f5_fd3ea26d2545.slice/crio-665befa699347cfeaced234853ed9dc9f00de651cfeaf166c1565dc02ed44d16\": RecentStats: unable to find data in memory cache]"
Oct 30 17:31:44.955454 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:44.955392    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-abe243-66559cf4d4-smdwk"]
Oct 30 17:31:45.696132 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:45.696096    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545" path="/var/lib/kubelet/pods/8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545/volumes"
Oct 30 17:31:52.959938 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:52.959901    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e60d-xn9fx"]
Oct 30 17:31:52.960597 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:52.960578    4433 cpu_manager.go:414] "containerMap: removing container" podUID="8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545" containerName="maas-api"
Oct 30 17:31:52.960597 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:52.960597    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545" containerName="maas-api"
Oct 30 17:31:52.960715 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:52.960705    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bd302f4-10c7-4e0b-a5f5-fd3ea26d2545" containerName="maas-api"
Oct 30 17:31:52.965600 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:52.965582    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e60d-xn9fx"
Oct 30 17:31:52.989519 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:52.989492    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e60d-xn9fx"]
Oct 30 17:31:53.056125 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.056089    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:31:53.056598 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.056579    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\""
Oct 30 17:31:53.061778 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.061750    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/123d0ace-03b1-4abe-a740-b373dea270f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-63e60d-xn9fx\" (UID: \"123d0ace-03b1-4abe-a740-b373dea270f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e60d-xn9fx"
Oct 30 17:31:53.061899 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.061790    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/123d0ace-03b1-4abe-a740-b373dea270f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-63e60d-xn9fx\" (UID: \"123d0ace-03b1-4abe-a740-b373dea270f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e60d-xn9fx"
Oct 30 17:31:53.062081 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.062059    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chswk\" (UniqueName: \"kubernetes.io/projected/123d0ace-03b1-4abe-a740-b373dea270f5-kube-api-access-chswk\") pod \"maas-api-revoke-keys-e2e-ipp-b-63e60d-xn9fx\" (UID: \"123d0ace-03b1-4abe-a740-b373dea270f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e60d-xn9fx"
Oct 30 17:31:53.163244 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.163209    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chswk\" (UniqueName: \"kubernetes.io/projected/123d0ace-03b1-4abe-a740-b373dea270f5-kube-api-access-chswk\") pod \"maas-api-revoke-keys-e2e-ipp-b-63e60d-xn9fx\" (UID: \"123d0ace-03b1-4abe-a740-b373dea270f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e60d-xn9fx"
Oct 30 17:31:53.163428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.163350    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/123d0ace-03b1-4abe-a740-b373dea270f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-63e60d-xn9fx\" (UID: \"123d0ace-03b1-4abe-a740-b373dea270f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e60d-xn9fx"
Oct 30 17:31:53.163428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.163379    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/123d0ace-03b1-4abe-a740-b373dea270f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-63e60d-xn9fx\" (UID: \"123d0ace-03b1-4abe-a740-b373dea270f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e60d-xn9fx"
Oct 30 17:31:53.164193 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.164158    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/123d0ace-03b1-4abe-a740-b373dea270f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-63e60d-xn9fx\" (UID: \"123d0ace-03b1-4abe-a740-b373dea270f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e60d-xn9fx"
Oct 30 17:31:53.165841 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.165817    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/123d0ace-03b1-4abe-a740-b373dea270f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-63e60d-xn9fx\" (UID: \"123d0ace-03b1-4abe-a740-b373dea270f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e60d-xn9fx"
Oct 30 17:31:53.186930 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.186904    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chswk\" (UniqueName: \"kubernetes.io/projected/123d0ace-03b1-4abe-a740-b373dea270f5-kube-api-access-chswk\") pod \"maas-api-revoke-keys-e2e-ipp-b-63e60d-xn9fx\" (UID: \"123d0ace-03b1-4abe-a740-b373dea270f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e60d-xn9fx"
Oct 30 17:31:53.280158 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.280072    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e60d-xn9fx"
Oct 30 17:31:53.460289 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.460241    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk" podUID="8d6ec843-3b36-407d-87bd-ae5a21472b61" containerName="main" containerID="cri-o://1511a123daec7817fbe00708c50d17b9b2a9bc57cfdc2784ceb8e054a3975e53" gracePeriod=45
Oct 30 17:31:53.584769 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.584744    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e60d-xn9fx"]
Oct 30 17:31:53.588012 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:31:53.587974    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod123d0ace_03b1_4abe_a740_b373dea270f5.slice/crio-3254e7ef515b407c43c3828a11f89fcca880d7825a9e193b4134213940fb5275 WatchSource:0}: Error finding container 3254e7ef515b407c43c3828a11f89fcca880d7825a9e193b4134213940fb5275: Status 404 returned error can't find the container with id 3254e7ef515b407c43c3828a11f89fcca880d7825a9e193b4134213940fb5275
Oct 30 17:31:53.767595 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.767566    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk"
Oct 30 17:31:53.877203 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.877124    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d6ec843-3b36-407d-87bd-ae5a21472b61-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d6ec843-3b36-407d-87bd-ae5a21472b61-tmp-dir\") pod \"8d6ec843-3b36-407d-87bd-ae5a21472b61\" (UID: \"8d6ec843-3b36-407d-87bd-ae5a21472b61\") "
Oct 30 17:31:53.877355 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.877214    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d6ec843-3b36-407d-87bd-ae5a21472b61-tls-certs\" (UniqueName: \"kubernetes.io/secret/8d6ec843-3b36-407d-87bd-ae5a21472b61-tls-certs\") pod \"8d6ec843-3b36-407d-87bd-ae5a21472b61\" (UID: \"8d6ec843-3b36-407d-87bd-ae5a21472b61\") "
Oct 30 17:31:53.877355 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.877235    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d6ec843-3b36-407d-87bd-ae5a21472b61-home\" (UniqueName: \"kubernetes.io/empty-dir/8d6ec843-3b36-407d-87bd-ae5a21472b61-home\") pod \"8d6ec843-3b36-407d-87bd-ae5a21472b61\" (UID: \"8d6ec843-3b36-407d-87bd-ae5a21472b61\") "
Oct 30 17:31:53.877355 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.877347    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d6ec843-3b36-407d-87bd-ae5a21472b61-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d6ec843-3b36-407d-87bd-ae5a21472b61-model-cache\") pod \"8d6ec843-3b36-407d-87bd-ae5a21472b61\" (UID: \"8d6ec843-3b36-407d-87bd-ae5a21472b61\") "
Oct 30 17:31:53.877553 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.877447    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6ec843-3b36-407d-87bd-ae5a21472b61-tmp-dir" pod "8d6ec843-3b36-407d-87bd-ae5a21472b61" (UID: "8d6ec843-3b36-407d-87bd-ae5a21472b61"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:31:53.877553 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.877455    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d6ec843-3b36-407d-87bd-ae5a21472b61-kube-api-access-5jwwx\" (UniqueName: \"kubernetes.io/projected/8d6ec843-3b36-407d-87bd-ae5a21472b61-kube-api-access-5jwwx\") pod \"8d6ec843-3b36-407d-87bd-ae5a21472b61\" (UID: \"8d6ec843-3b36-407d-87bd-ae5a21472b61\") "
Oct 30 17:31:53.877664 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.877565    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d6ec843-3b36-407d-87bd-ae5a21472b61-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d6ec843-3b36-407d-87bd-ae5a21472b61-dshm\") pod \"8d6ec843-3b36-407d-87bd-ae5a21472b61\" (UID: \"8d6ec843-3b36-407d-87bd-ae5a21472b61\") "
Oct 30 17:31:53.877710 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.877653    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6ec843-3b36-407d-87bd-ae5a21472b61-home" pod "8d6ec843-3b36-407d-87bd-ae5a21472b61" (UID: "8d6ec843-3b36-407d-87bd-ae5a21472b61"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:31:53.877710 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.877670    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6ec843-3b36-407d-87bd-ae5a21472b61-model-cache" pod "8d6ec843-3b36-407d-87bd-ae5a21472b61" (UID: "8d6ec843-3b36-407d-87bd-ae5a21472b61"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:31:53.878563 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.878537    4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d6ec843-3b36-407d-87bd-ae5a21472b61-home\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:31:53.878704 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.878566    4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d6ec843-3b36-407d-87bd-ae5a21472b61-model-cache\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:31:53.878704 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.878583    4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d6ec843-3b36-407d-87bd-ae5a21472b61-tmp-dir\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:31:53.879730 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.879704    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d6ec843-3b36-407d-87bd-ae5a21472b61-tls-certs" pod "8d6ec843-3b36-407d-87bd-ae5a21472b61" (UID: "8d6ec843-3b36-407d-87bd-ae5a21472b61"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:31:53.879821 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.879739    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d6ec843-3b36-407d-87bd-ae5a21472b61-kube-api-access-5jwwx" pod "8d6ec843-3b36-407d-87bd-ae5a21472b61" (UID: "8d6ec843-3b36-407d-87bd-ae5a21472b61"). InnerVolumeSpecName "kube-api-access-5jwwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:31:53.879821 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.879763    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6ec843-3b36-407d-87bd-ae5a21472b61-dshm" pod "8d6ec843-3b36-407d-87bd-ae5a21472b61" (UID: "8d6ec843-3b36-407d-87bd-ae5a21472b61"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:31:53.952053 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.952009    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e60d-xn9fx" event={"ID":"123d0ace-03b1-4abe-a740-b373dea270f5","Type":"ContainerStarted","Data":"3254e7ef515b407c43c3828a11f89fcca880d7825a9e193b4134213940fb5275"}
Oct 30 17:31:53.953590 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.953561    4433 generic.go:356] "Generic (PLEG): container finished" podID="8d6ec843-3b36-407d-87bd-ae5a21472b61" containerID="1511a123daec7817fbe00708c50d17b9b2a9bc57cfdc2784ceb8e054a3975e53" exitCode=0
Oct 30 17:31:53.953728 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.953610    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk"
Oct 30 17:31:53.953728 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.953626    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk" event={"ID":"8d6ec843-3b36-407d-87bd-ae5a21472b61","Type":"ContainerDied","Data":"1511a123daec7817fbe00708c50d17b9b2a9bc57cfdc2784ceb8e054a3975e53"}
Oct 30 17:31:53.953728 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.953662    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk" event={"ID":"8d6ec843-3b36-407d-87bd-ae5a21472b61","Type":"ContainerDied","Data":"b3b89cba850bd86e96f10bddbcdda9b4bacecd298dc6eea4087883f35285c91d"}
Oct 30 17:31:53.953728 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.953683    4433 scope.go:122] "RemoveContainer" containerID="1511a123daec7817fbe00708c50d17b9b2a9bc57cfdc2784ceb8e054a3975e53"
Oct 30 17:31:53.969628 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.969607    4433 scope.go:122] "RemoveContainer" containerID="1511a123daec7817fbe00708c50d17b9b2a9bc57cfdc2784ceb8e054a3975e53"
Oct 30 17:31:53.969962 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:31:53.969942    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1511a123daec7817fbe00708c50d17b9b2a9bc57cfdc2784ceb8e054a3975e53\": container with ID starting with 1511a123daec7817fbe00708c50d17b9b2a9bc57cfdc2784ceb8e054a3975e53 not found: ID does not exist" containerID="1511a123daec7817fbe00708c50d17b9b2a9bc57cfdc2784ceb8e054a3975e53"
Oct 30 17:31:53.970022 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.969976    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1511a123daec7817fbe00708c50d17b9b2a9bc57cfdc2784ceb8e054a3975e53"} err="failed to get container status \"1511a123daec7817fbe00708c50d17b9b2a9bc57cfdc2784ceb8e054a3975e53\": rpc error: code = NotFound desc = could not find container \"1511a123daec7817fbe00708c50d17b9b2a9bc57cfdc2784ceb8e054a3975e53\": container with ID starting with 1511a123daec7817fbe00708c50d17b9b2a9bc57cfdc2784ceb8e054a3975e53 not found: ID does not exist"
Oct 30 17:31:53.980088 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.980067    4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d6ec843-3b36-407d-87bd-ae5a21472b61-tls-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:31:53.980088 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.980090    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jwwx\" (UniqueName: \"kubernetes.io/projected/8d6ec843-3b36-407d-87bd-ae5a21472b61-kube-api-access-5jwwx\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:31:53.980210 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:53.980101    4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d6ec843-3b36-407d-87bd-ae5a21472b61-dshm\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:31:54.005641 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:54.005607    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk"]
Oct 30 17:31:54.010857 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:54.010835    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk"]
Oct 30 17:31:54.680436 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:54.678341    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-469ad7/auth-test-model-469ad7-kserve-6d88d6fdbf-pdrkk" podUID="8d6ec843-3b36-407d-87bd-ae5a21472b61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.151:8000/ready\": context deadline exceeded"
Oct 30 17:31:54.962076 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:54.961989    4433 generic.go:356] "Generic (PLEG): container finished" podID="123d0ace-03b1-4abe-a740-b373dea270f5" containerID="f9e2987d8d380f9694fe0b1d9ebad9da12e532595279bf5d5068d7b9489e6f5b" exitCode=0
Oct 30 17:31:54.962235 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:54.962109    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e60d-xn9fx" event={"ID":"123d0ace-03b1-4abe-a740-b373dea270f5","Type":"ContainerDied","Data":"f9e2987d8d380f9694fe0b1d9ebad9da12e532595279bf5d5068d7b9489e6f5b"}
Oct 30 17:31:55.696847 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:55.696814    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d6ec843-3b36-407d-87bd-ae5a21472b61" path="/var/lib/kubelet/pods/8d6ec843-3b36-407d-87bd-ae5a21472b61/volumes"
Oct 30 17:31:56.140735 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:56.140708    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e60d-xn9fx"
Oct 30 17:31:56.314918 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:56.314834    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/123d0ace-03b1-4abe-a740-b373dea270f5-kube-api-access-chswk\" (UniqueName: \"kubernetes.io/projected/123d0ace-03b1-4abe-a740-b373dea270f5-kube-api-access-chswk\") pod \"123d0ace-03b1-4abe-a740-b373dea270f5\" (UID: \"123d0ace-03b1-4abe-a740-b373dea270f5\") "
Oct 30 17:31:56.315322 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:56.315295    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/123d0ace-03b1-4abe-a740-b373dea270f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/123d0ace-03b1-4abe-a740-b373dea270f5-maas-api-openshift-ca\") pod \"123d0ace-03b1-4abe-a740-b373dea270f5\" (UID: \"123d0ace-03b1-4abe-a740-b373dea270f5\") "
Oct 30 17:31:56.315686 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:56.315644    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/123d0ace-03b1-4abe-a740-b373dea270f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/123d0ace-03b1-4abe-a740-b373dea270f5-maas-api-xks-ca\") pod \"123d0ace-03b1-4abe-a740-b373dea270f5\" (UID: \"123d0ace-03b1-4abe-a740-b373dea270f5\") "
Oct 30 17:31:56.316980 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:56.316953    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/123d0ace-03b1-4abe-a740-b373dea270f5-maas-api-openshift-ca" pod "123d0ace-03b1-4abe-a740-b373dea270f5" (UID: "123d0ace-03b1-4abe-a740-b373dea270f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:31:56.318830 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:56.318807    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/123d0ace-03b1-4abe-a740-b373dea270f5-kube-api-access-chswk" pod "123d0ace-03b1-4abe-a740-b373dea270f5" (UID: "123d0ace-03b1-4abe-a740-b373dea270f5"). InnerVolumeSpecName "kube-api-access-chswk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:31:56.318941 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:56.318807    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/123d0ace-03b1-4abe-a740-b373dea270f5-maas-api-xks-ca" pod "123d0ace-03b1-4abe-a740-b373dea270f5" (UID: "123d0ace-03b1-4abe-a740-b373dea270f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:31:56.418124 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:56.418078    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/123d0ace-03b1-4abe-a740-b373dea270f5-maas-api-xks-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:31:56.418124 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:56.418110    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chswk\" (UniqueName: \"kubernetes.io/projected/123d0ace-03b1-4abe-a740-b373dea270f5-kube-api-access-chswk\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:31:56.418124 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:56.418122    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/123d0ace-03b1-4abe-a740-b373dea270f5-maas-api-openshift-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:31:56.979517 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:31:56.979480    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:31:56.981951 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:56.981926    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e60d-xn9fx" event={"ID":"123d0ace-03b1-4abe-a740-b373dea270f5","Type":"ContainerDied","Data":"3254e7ef515b407c43c3828a11f89fcca880d7825a9e193b4134213940fb5275"}
Oct 30 17:31:56.982064 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:56.981960    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e60d-xn9fx"
Oct 30 17:31:56.982064 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:31:56.981969    4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3254e7ef515b407c43c3828a11f89fcca880d7825a9e193b4134213940fb5275"
Oct 30 17:32:04.995370 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:04.995329    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"]
Oct 30 17:32:04.996394 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:04.996366    4433 cpu_manager.go:414] "containerMap: removing container" podUID="123d0ace-03b1-4abe-a740-b373dea270f5" containerName="revoke-keys"
Oct 30 17:32:04.996526 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:04.996395    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="123d0ace-03b1-4abe-a740-b373dea270f5" containerName="revoke-keys"
Oct 30 17:32:04.996526 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:04.996448    4433 cpu_manager.go:414] "containerMap: removing container" podUID="8d6ec843-3b36-407d-87bd-ae5a21472b61" containerName="main"
Oct 30 17:32:04.996526 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:04.996456    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d6ec843-3b36-407d-87bd-ae5a21472b61" containerName="main"
Oct 30 17:32:04.996704 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:04.996677    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="123d0ace-03b1-4abe-a740-b373dea270f5" containerName="revoke-keys"
Oct 30 17:32:04.996704 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:04.996698    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d6ec843-3b36-407d-87bd-ae5a21472b61" containerName="main"
Oct 30 17:32:05.000863 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.000837    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:32:05.007871 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.007839    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:32:05.025233 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.025205    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"]
Oct 30 17:32:05.026599 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.026512    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-a949e7-openshift-default-dockercfg-zp2cm\""
Oct 30 17:32:05.067511 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.067474    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b410f33-6553-4f8d-972f-b21558c4db1a-istio-podinfo\") pod \"e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") " pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:32:05.067511 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.067514    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b410f33-6553-4f8d-972f-b21558c4db1a-istio-token\") pod \"e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") " pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:32:05.067757 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.067609    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b86wj\" (UniqueName: \"kubernetes.io/projected/8b410f33-6553-4f8d-972f-b21558c4db1a-kube-api-access-b86wj\") pod \"e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") " pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:32:05.067757 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.067709    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b410f33-6553-4f8d-972f-b21558c4db1a-istio-envoy\") pod \"e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") " pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:32:05.067877 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.067753    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b410f33-6553-4f8d-972f-b21558c4db1a-workload-socket\") pod \"e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") " pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:32:05.067937 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.067912    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b410f33-6553-4f8d-972f-b21558c4db1a-istio-data\") pod \"e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") " pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:32:05.067996 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.067960    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b410f33-6553-4f8d-972f-b21558c4db1a-istiod-ca-cert\") pod \"e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") " pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:32:05.068071 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.068046    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b410f33-6553-4f8d-972f-b21558c4db1a-workload-certs\") pod \"e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") " pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:32:05.068376 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.068346    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b410f33-6553-4f8d-972f-b21558c4db1a-credential-socket\") pod \"e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") " pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:32:05.203004 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.201923    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b410f33-6553-4f8d-972f-b21558c4db1a-credential-socket\") pod \"e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") " pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:32:05.203004 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.202515    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b410f33-6553-4f8d-972f-b21558c4db1a-credential-socket\") pod \"e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") " pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:32:05.203004 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.202685    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b410f33-6553-4f8d-972f-b21558c4db1a-istio-podinfo\") pod \"e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") " pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:32:05.203004 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.202728    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b410f33-6553-4f8d-972f-b21558c4db1a-istio-token\") pod \"e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") " pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:32:05.203004 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.202809    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b86wj\" (UniqueName: \"kubernetes.io/projected/8b410f33-6553-4f8d-972f-b21558c4db1a-kube-api-access-b86wj\") pod \"e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") " pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:32:05.203004 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.202894    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b410f33-6553-4f8d-972f-b21558c4db1a-istio-envoy\") pod \"e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") " pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:32:05.203004 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.202937    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b410f33-6553-4f8d-972f-b21558c4db1a-workload-socket\") pod \"e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") " pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:32:05.203547 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.203033    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b410f33-6553-4f8d-972f-b21558c4db1a-istio-data\") pod \"e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") " pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:32:05.203547 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.203074    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b410f33-6553-4f8d-972f-b21558c4db1a-istiod-ca-cert\") pod \"e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") " pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:32:05.203547 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.203136    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b410f33-6553-4f8d-972f-b21558c4db1a-workload-certs\") pod \"e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") " pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:32:05.203740 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.203715    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b410f33-6553-4f8d-972f-b21558c4db1a-workload-certs\") pod \"e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") " pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:32:05.220250 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.217762    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b410f33-6553-4f8d-972f-b21558c4db1a-istio-data\") pod \"e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") " pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:32:05.220250 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.217899    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b410f33-6553-4f8d-972f-b21558c4db1a-workload-socket\") pod \"e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") " pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:32:05.220250 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.219040    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b410f33-6553-4f8d-972f-b21558c4db1a-istiod-ca-cert\") pod \"e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") " pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:32:05.237436 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.231509    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b410f33-6553-4f8d-972f-b21558c4db1a-istio-podinfo\") pod \"e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") " pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:32:05.238354 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.238326    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b410f33-6553-4f8d-972f-b21558c4db1a-istio-envoy\") pod \"e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") " pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:32:05.245267 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.243896    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b410f33-6553-4f8d-972f-b21558c4db1a-istio-token\") pod \"e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") " pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:32:05.267468 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.257358    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b86wj\" (UniqueName: \"kubernetes.io/projected/8b410f33-6553-4f8d-972f-b21558c4db1a-kube-api-access-b86wj\") pod \"e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") " pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:32:05.324426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.323667    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:32:05.845871 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.845838    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"]
Oct 30 17:32:05.846631 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:32:05.846592    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b410f33_6553_4f8d_972f_b21558c4db1a.slice/crio-d9e523a180dd947792761ad9a52f79b9b451c8c29b721479e05b63181f9190a4 WatchSource:0}: Error finding container d9e523a180dd947792761ad9a52f79b9b451c8c29b721479e05b63181f9190a4: Status 404 returned error can't find the container with id d9e523a180dd947792761ad9a52f79b9b451c8c29b721479e05b63181f9190a4
Oct 30 17:32:05.850469 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.850387    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:32:05.850563 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.850523    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:32:05.850606 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:05.850568    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:32:06.098849 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:06.098688    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m" event={"ID":"8b410f33-6553-4f8d-972f-b21558c4db1a","Type":"ContainerStarted","Data":"cc9c266c294d99efd5fd767e042e7c54ef42b46fe7abd8d39d868139a859c176"}
Oct 30 17:32:06.098849 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:06.098762    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m" event={"ID":"8b410f33-6553-4f8d-972f-b21558c4db1a","Type":"ContainerStarted","Data":"d9e523a180dd947792761ad9a52f79b9b451c8c29b721479e05b63181f9190a4"}
Oct 30 17:32:06.170104 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:06.170042    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m" podStartSLOduration=2.170021082 podStartE2EDuration="2.170021082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:32:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:32:06.164245344 +0000 UTC m=+2702.599147569" watchObservedRunningTime="2026-10-30 17:32:06.170021082 +0000 UTC m=+2702.604923288"
Oct 30 17:32:06.173544 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:06.173516    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3260f8-f9c947b48-xn892"]
Oct 30 17:32:06.327869 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:06.324970    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:32:06.327869 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:06.326469    4433 patch_prober.go:29] interesting pod/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 17:32:06.327869 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:06.326517    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m" podUID="8b410f33-6553-4f8d-972f-b21558c4db1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 17:32:07.625433 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:07.623975    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:07.647431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:07.644928    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:32:07.682313 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:07.671839    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:07.704426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:07.694513    4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:07.704426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:07.694596    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm" podUID="bf43bf3d-f126-40a1-a24f-10214a85ceb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:08.014452 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:08.012503    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:08.104428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:08.103504    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:32:08.172696 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:08.172667    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:32:08.175595 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:08.175572    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:32:11.174497 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:11.174458    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-f9c947b48-xn892" podUID="d0a8d678-3fd9-4ce9-9e9f-f123f42dda10" containerName="payload-processing" containerID="cri-o://104e854abe6ebc1f9025271dbd5b9f771bbbde6aa25f4cd031879a71bb0cb5cd" gracePeriod=25
Oct 30 17:32:11.472238 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:11.472205    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-f9c947b48-xn892"
Oct 30 17:32:11.624968 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:11.624929    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0a8d678-3fd9-4ce9-9e9f-f123f42dda10-kube-api-access-5grmq\" (UniqueName: \"kubernetes.io/projected/d0a8d678-3fd9-4ce9-9e9f-f123f42dda10-kube-api-access-5grmq\") pod \"d0a8d678-3fd9-4ce9-9e9f-f123f42dda10\" (UID: \"d0a8d678-3fd9-4ce9-9e9f-f123f42dda10\") "
Oct 30 17:32:11.625529 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:11.625502    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0a8d678-3fd9-4ce9-9e9f-f123f42dda10-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a8d678-3fd9-4ce9-9e9f-f123f42dda10-plugins-config-volume\") pod \"d0a8d678-3fd9-4ce9-9e9f-f123f42dda10\" (UID: \"d0a8d678-3fd9-4ce9-9e9f-f123f42dda10\") "
Oct 30 17:32:11.640674 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:11.640632    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a8d678-3fd9-4ce9-9e9f-f123f42dda10-plugins-config-volume" pod "d0a8d678-3fd9-4ce9-9e9f-f123f42dda10" (UID: "d0a8d678-3fd9-4ce9-9e9f-f123f42dda10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:32:11.734104 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:11.733995    4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a8d678-3fd9-4ce9-9e9f-f123f42dda10-plugins-config-volume\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:32:11.753397 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:11.753361    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a8d678-3fd9-4ce9-9e9f-f123f42dda10-kube-api-access-5grmq" pod "d0a8d678-3fd9-4ce9-9e9f-f123f42dda10" (UID: "d0a8d678-3fd9-4ce9-9e9f-f123f42dda10"). InnerVolumeSpecName "kube-api-access-5grmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:32:11.863432 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:11.861844    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5grmq\" (UniqueName: \"kubernetes.io/projected/d0a8d678-3fd9-4ce9-9e9f-f123f42dda10-kube-api-access-5grmq\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:32:12.135570 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:12.135380    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a949e7-7cff8d95bb-9r6gv"]
Oct 30 17:32:12.136902 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:12.136857    4433 cpu_manager.go:414] "containerMap: removing container" podUID="d0a8d678-3fd9-4ce9-9e9f-f123f42dda10" containerName="payload-processing"
Oct 30 17:32:12.137093 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:12.137073    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0a8d678-3fd9-4ce9-9e9f-f123f42dda10" containerName="payload-processing"
Oct 30 17:32:12.137377 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:12.137356    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0a8d678-3fd9-4ce9-9e9f-f123f42dda10" containerName="payload-processing"
Oct 30 17:32:12.149675 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:12.149647    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-7cff8d95bb-9r6gv"
Oct 30 17:32:12.188535 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:12.188384    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7894b\" (UniqueName: \"kubernetes.io/projected/db411383-a64b-4614-b496-abe53f6cc5c0-kube-api-access-7894b\") pod \"payload-processing-e2e-ipp-a-a949e7-7cff8d95bb-9r6gv\" (UID: \"db411383-a64b-4614-b496-abe53f6cc5c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-7cff8d95bb-9r6gv"
Oct 30 17:32:12.189332 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:12.189302    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db411383-a64b-4614-b496-abe53f6cc5c0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a949e7-7cff8d95bb-9r6gv\" (UID: \"db411383-a64b-4614-b496-abe53f6cc5c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-7cff8d95bb-9r6gv"
Oct 30 17:32:12.191918 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:12.191280    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a949e7\""
Oct 30 17:32:12.191918 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:12.191431    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a949e7-dockercfg-6bbsx\""
Oct 30 17:32:12.251564 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:12.251526    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a949e7-7cff8d95bb-9r6gv"]
Oct 30 17:32:12.291100 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:12.291055    4433 generic.go:356] "Generic (PLEG): container finished" podID="d0a8d678-3fd9-4ce9-9e9f-f123f42dda10" containerID="104e854abe6ebc1f9025271dbd5b9f771bbbde6aa25f4cd031879a71bb0cb5cd" exitCode=0
Oct 30 17:32:12.291804 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:12.291192    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db411383-a64b-4614-b496-abe53f6cc5c0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a949e7-7cff8d95bb-9r6gv\" (UID: \"db411383-a64b-4614-b496-abe53f6cc5c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-7cff8d95bb-9r6gv"
Oct 30 17:32:12.291804 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:12.291374    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7894b\" (UniqueName: \"kubernetes.io/projected/db411383-a64b-4614-b496-abe53f6cc5c0-kube-api-access-7894b\") pod \"payload-processing-e2e-ipp-a-a949e7-7cff8d95bb-9r6gv\" (UID: \"db411383-a64b-4614-b496-abe53f6cc5c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-7cff8d95bb-9r6gv"
Oct 30 17:32:12.291804 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:12.291533    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-f9c947b48-xn892" event={"ID":"d0a8d678-3fd9-4ce9-9e9f-f123f42dda10","Type":"ContainerDied","Data":"104e854abe6ebc1f9025271dbd5b9f771bbbde6aa25f4cd031879a71bb0cb5cd"}
Oct 30 17:32:12.291804 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:12.291601    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-f9c947b48-xn892" event={"ID":"d0a8d678-3fd9-4ce9-9e9f-f123f42dda10","Type":"ContainerDied","Data":"9b5d1f319bd1f3b168349a038755e43e4411518eb1581e2a29b69025fefd1dc5"}
Oct 30 17:32:12.291804 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:12.291648    4433 scope.go:122] "RemoveContainer" containerID="104e854abe6ebc1f9025271dbd5b9f771bbbde6aa25f4cd031879a71bb0cb5cd"
Oct 30 17:32:12.293341 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:12.293311    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-f9c947b48-xn892"
Oct 30 17:32:12.309795 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:12.309595    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db411383-a64b-4614-b496-abe53f6cc5c0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a949e7-7cff8d95bb-9r6gv\" (UID: \"db411383-a64b-4614-b496-abe53f6cc5c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-7cff8d95bb-9r6gv"
Oct 30 17:32:12.320022 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:12.320000    4433 scope.go:122] "RemoveContainer" containerID="104e854abe6ebc1f9025271dbd5b9f771bbbde6aa25f4cd031879a71bb0cb5cd"
Oct 30 17:32:12.320515 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:32:12.320487    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"104e854abe6ebc1f9025271dbd5b9f771bbbde6aa25f4cd031879a71bb0cb5cd\": container with ID starting with 104e854abe6ebc1f9025271dbd5b9f771bbbde6aa25f4cd031879a71bb0cb5cd not found: ID does not exist" containerID="104e854abe6ebc1f9025271dbd5b9f771bbbde6aa25f4cd031879a71bb0cb5cd"
Oct 30 17:32:12.320657 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:12.320635    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"104e854abe6ebc1f9025271dbd5b9f771bbbde6aa25f4cd031879a71bb0cb5cd"} err="failed to get container status \"104e854abe6ebc1f9025271dbd5b9f771bbbde6aa25f4cd031879a71bb0cb5cd\": rpc error: code = NotFound desc = could not find container \"104e854abe6ebc1f9025271dbd5b9f771bbbde6aa25f4cd031879a71bb0cb5cd\": container with ID starting with 104e854abe6ebc1f9025271dbd5b9f771bbbde6aa25f4cd031879a71bb0cb5cd not found: ID does not exist"
Oct 30 17:32:12.329266 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:12.329243    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7894b\" (UniqueName: \"kubernetes.io/projected/db411383-a64b-4614-b496-abe53f6cc5c0-kube-api-access-7894b\") pod \"payload-processing-e2e-ipp-a-a949e7-7cff8d95bb-9r6gv\" (UID: \"db411383-a64b-4614-b496-abe53f6cc5c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-7cff8d95bb-9r6gv"
Oct 30 17:32:12.358290 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:12.358255    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3260f8-f9c947b48-xn892"]
Oct 30 17:32:12.364453 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:12.364420    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3260f8-f9c947b48-xn892"]
Oct 30 17:32:12.580771 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:12.580735    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-7cff8d95bb-9r6gv"
Oct 30 17:32:12.888971 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:12.888945    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a949e7-7cff8d95bb-9r6gv"]
Oct 30 17:32:12.890251 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:32:12.890228    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb411383_a64b_4614_b496_abe53f6cc5c0.slice/crio-0a85a2326d94fa0caac06c6f1c9c944fe5b1a2da7ec071728c5d73ae9da07c09 WatchSource:0}: Error finding container 0a85a2326d94fa0caac06c6f1c9c944fe5b1a2da7ec071728c5d73ae9da07c09: Status 404 returned error can't find the container with id 0a85a2326d94fa0caac06c6f1c9c944fe5b1a2da7ec071728c5d73ae9da07c09
Oct 30 17:32:13.300690 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:13.300657    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-7cff8d95bb-9r6gv" event={"ID":"db411383-a64b-4614-b496-abe53f6cc5c0","Type":"ContainerStarted","Data":"48c21080dfcbc80ed5be0caa5bbb0ca19efe4abdbcb32d66a9e06011c8c48f50"}
Oct 30 17:32:13.301100 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:13.300706    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-7cff8d95bb-9r6gv" event={"ID":"db411383-a64b-4614-b496-abe53f6cc5c0","Type":"ContainerStarted","Data":"0a85a2326d94fa0caac06c6f1c9c944fe5b1a2da7ec071728c5d73ae9da07c09"}
Oct 30 17:32:13.301100 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:13.300798    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-7cff8d95bb-9r6gv"
Oct 30 17:32:13.347362 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:13.347305    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-7cff8d95bb-9r6gv" podStartSLOduration=1.347287447 podStartE2EDuration="1.347287447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:32:13.343995337 +0000 UTC m=+2709.778897543" watchObservedRunningTime="2026-10-30 17:32:13.347287447 +0000 UTC m=+2709.782189653"
Oct 30 17:32:13.697676 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:13.697647    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0a8d678-3fd9-4ce9-9e9f-f123f42dda10" path="/var/lib/kubelet/pods/d0a8d678-3fd9-4ce9-9e9f-f123f42dda10/volumes"
Oct 30 17:32:20.248548 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:20.248512    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e60d-xn9fx"]
Oct 30 17:32:20.258677 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:20.258650    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e60d-xn9fx"]
Oct 30 17:32:21.697483 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:21.697446    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="123d0ace-03b1-4abe-a740-b373dea270f5" path="/var/lib/kubelet/pods/123d0ace-03b1-4abe-a740-b373dea270f5/volumes"
Oct 30 17:32:24.310154 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:24.310115    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-7cff8d95bb-9r6gv"
Oct 30 17:32:26.978957 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:32:26.978919    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:32:28.714229 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:28.714192    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-987dfc-b6gf5"]
Oct 30 17:32:28.720157 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:28.720133    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-987dfc-b6gf5"
Oct 30 17:32:28.738591 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:28.738560    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-987dfc-b6gf5"]
Oct 30 17:32:28.813384 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:28.813349    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:32:28.813384 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:28.813369    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\""
Oct 30 17:32:28.851525 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:28.851494    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e2cce20-83b4-4883-8fd3-4dba0cb837ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-987dfc-b6gf5\" (UID: \"6e2cce20-83b4-4883-8fd3-4dba0cb837ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-987dfc-b6gf5"
Oct 30 17:32:28.851807 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:28.851783    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e2cce20-83b4-4883-8fd3-4dba0cb837ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-987dfc-b6gf5\" (UID: \"6e2cce20-83b4-4883-8fd3-4dba0cb837ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-987dfc-b6gf5"
Oct 30 17:32:28.851853 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:28.851842    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xqd4\" (UniqueName: \"kubernetes.io/projected/6e2cce20-83b4-4883-8fd3-4dba0cb837ff-kube-api-access-9xqd4\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-987dfc-b6gf5\" (UID: \"6e2cce20-83b4-4883-8fd3-4dba0cb837ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-987dfc-b6gf5"
Oct 30 17:32:28.953212 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:28.953167    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e2cce20-83b4-4883-8fd3-4dba0cb837ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-987dfc-b6gf5\" (UID: \"6e2cce20-83b4-4883-8fd3-4dba0cb837ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-987dfc-b6gf5"
Oct 30 17:32:28.953520 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:28.953500    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e2cce20-83b4-4883-8fd3-4dba0cb837ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-987dfc-b6gf5\" (UID: \"6e2cce20-83b4-4883-8fd3-4dba0cb837ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-987dfc-b6gf5"
Oct 30 17:32:28.953638 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:28.953568    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xqd4\" (UniqueName: \"kubernetes.io/projected/6e2cce20-83b4-4883-8fd3-4dba0cb837ff-kube-api-access-9xqd4\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-987dfc-b6gf5\" (UID: \"6e2cce20-83b4-4883-8fd3-4dba0cb837ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-987dfc-b6gf5"
Oct 30 17:32:28.953882 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:28.953851    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e2cce20-83b4-4883-8fd3-4dba0cb837ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-987dfc-b6gf5\" (UID: \"6e2cce20-83b4-4883-8fd3-4dba0cb837ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-987dfc-b6gf5"
Oct 30 17:32:28.955883 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:28.955853    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e2cce20-83b4-4883-8fd3-4dba0cb837ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-987dfc-b6gf5\" (UID: \"6e2cce20-83b4-4883-8fd3-4dba0cb837ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-987dfc-b6gf5"
Oct 30 17:32:28.976080 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:28.976021    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xqd4\" (UniqueName: \"kubernetes.io/projected/6e2cce20-83b4-4883-8fd3-4dba0cb837ff-kube-api-access-9xqd4\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-987dfc-b6gf5\" (UID: \"6e2cce20-83b4-4883-8fd3-4dba0cb837ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-987dfc-b6gf5"
Oct 30 17:32:29.034387 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:29.034353    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-987dfc-b6gf5"
Oct 30 17:32:29.342876 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:29.342852    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-987dfc-b6gf5"]
Oct 30 17:32:29.345334 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:32:29.345309    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e2cce20_83b4_4883_8fd3_4dba0cb837ff.slice/crio-0d455442545e554e684fce95bc46a49aa22f1dbb98934f726d2a7871423a9cb5 WatchSource:0}: Error finding container 0d455442545e554e684fce95bc46a49aa22f1dbb98934f726d2a7871423a9cb5: Status 404 returned error can't find the container with id 0d455442545e554e684fce95bc46a49aa22f1dbb98934f726d2a7871423a9cb5
Oct 30 17:32:29.509024 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:29.508989    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-987dfc-b6gf5" event={"ID":"6e2cce20-83b4-4883-8fd3-4dba0cb837ff","Type":"ContainerStarted","Data":"0d455442545e554e684fce95bc46a49aa22f1dbb98934f726d2a7871423a9cb5"}
Oct 30 17:32:30.517628 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:30.517585    4433 generic.go:356] "Generic (PLEG): container finished" podID="6e2cce20-83b4-4883-8fd3-4dba0cb837ff" containerID="be2aae9144715a73a9253392ac6ac738029d9a35b9f688d272bb8b5389660b98" exitCode=0
Oct 30 17:32:30.518035 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:30.517665    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-987dfc-b6gf5" event={"ID":"6e2cce20-83b4-4883-8fd3-4dba0cb837ff","Type":"ContainerDied","Data":"be2aae9144715a73a9253392ac6ac738029d9a35b9f688d272bb8b5389660b98"}
Oct 30 17:32:31.796998 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:31.796924    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-987dfc-b6gf5"
Oct 30 17:32:31.849797 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:31.849759    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e2cce20-83b4-4883-8fd3-4dba0cb837ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e2cce20-83b4-4883-8fd3-4dba0cb837ff-maas-api-openshift-ca\") pod \"6e2cce20-83b4-4883-8fd3-4dba0cb837ff\" (UID: \"6e2cce20-83b4-4883-8fd3-4dba0cb837ff\") "
Oct 30 17:32:31.850020 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:31.849919    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e2cce20-83b4-4883-8fd3-4dba0cb837ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e2cce20-83b4-4883-8fd3-4dba0cb837ff-maas-api-xks-ca\") pod \"6e2cce20-83b4-4883-8fd3-4dba0cb837ff\" (UID: \"6e2cce20-83b4-4883-8fd3-4dba0cb837ff\") "
Oct 30 17:32:31.850120 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:31.850042    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e2cce20-83b4-4883-8fd3-4dba0cb837ff-kube-api-access-9xqd4\" (UniqueName: \"kubernetes.io/projected/6e2cce20-83b4-4883-8fd3-4dba0cb837ff-kube-api-access-9xqd4\") pod \"6e2cce20-83b4-4883-8fd3-4dba0cb837ff\" (UID: \"6e2cce20-83b4-4883-8fd3-4dba0cb837ff\") "
Oct 30 17:32:31.850293 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:31.850237    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e2cce20-83b4-4883-8fd3-4dba0cb837ff-maas-api-openshift-ca" pod "6e2cce20-83b4-4883-8fd3-4dba0cb837ff" (UID: "6e2cce20-83b4-4883-8fd3-4dba0cb837ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:32:31.851033 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:31.851003    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e2cce20-83b4-4883-8fd3-4dba0cb837ff-maas-api-openshift-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:32:31.852890 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:31.852852    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e2cce20-83b4-4883-8fd3-4dba0cb837ff-kube-api-access-9xqd4" pod "6e2cce20-83b4-4883-8fd3-4dba0cb837ff" (UID: "6e2cce20-83b4-4883-8fd3-4dba0cb837ff"). InnerVolumeSpecName "kube-api-access-9xqd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:32:31.853911 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:31.853884    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e2cce20-83b4-4883-8fd3-4dba0cb837ff-maas-api-xks-ca" pod "6e2cce20-83b4-4883-8fd3-4dba0cb837ff" (UID: "6e2cce20-83b4-4883-8fd3-4dba0cb837ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:32:31.952305 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:31.952269    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xqd4\" (UniqueName: \"kubernetes.io/projected/6e2cce20-83b4-4883-8fd3-4dba0cb837ff-kube-api-access-9xqd4\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:32:31.952305 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:31.952301    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e2cce20-83b4-4883-8fd3-4dba0cb837ff-maas-api-xks-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:32:32.535042 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:32.535008    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-987dfc-b6gf5"
Oct 30 17:32:32.535231 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:32.535040    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-987dfc-b6gf5" event={"ID":"6e2cce20-83b4-4883-8fd3-4dba0cb837ff","Type":"ContainerDied","Data":"0d455442545e554e684fce95bc46a49aa22f1dbb98934f726d2a7871423a9cb5"}
Oct 30 17:32:32.535231 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:32.535091    4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d455442545e554e684fce95bc46a49aa22f1dbb98934f726d2a7871423a9cb5"
Oct 30 17:32:34.632908 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:34.632864    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2d985-bw87p"]
Oct 30 17:32:34.633757 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:34.633733    4433 cpu_manager.go:414] "containerMap: removing container" podUID="6e2cce20-83b4-4883-8fd3-4dba0cb837ff" containerName="revoke-keys"
Oct 30 17:32:34.633757 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:34.633758    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e2cce20-83b4-4883-8fd3-4dba0cb837ff" containerName="revoke-keys"
Oct 30 17:32:34.633974 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:34.633959    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e2cce20-83b4-4883-8fd3-4dba0cb837ff" containerName="revoke-keys"
Oct 30 17:32:34.637509 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:34.637486    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2d985-bw87p"
Oct 30 17:32:34.671468 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:34.661175    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2d985-bw87p"]
Oct 30 17:32:34.677915 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:34.677885    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:32:34.678065 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:34.677953    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\""
Oct 30 17:32:34.701266 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:34.701234    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp5tf\" (UniqueName: \"kubernetes.io/projected/5fa2b418-d47e-4f0c-b0f5-36aa9dd14030-kube-api-access-hp5tf\") pod \"maas-api-revoke-keys-e2e-ipp-a-d2d985-bw87p\" (UID: \"5fa2b418-d47e-4f0c-b0f5-36aa9dd14030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2d985-bw87p"
Oct 30 17:32:34.701444 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:34.701274    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fa2b418-d47e-4f0c-b0f5-36aa9dd14030-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d2d985-bw87p\" (UID: \"5fa2b418-d47e-4f0c-b0f5-36aa9dd14030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2d985-bw87p"
Oct 30 17:32:34.701520 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:34.701478    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fa2b418-d47e-4f0c-b0f5-36aa9dd14030-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d2d985-bw87p\" (UID: \"5fa2b418-d47e-4f0c-b0f5-36aa9dd14030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2d985-bw87p"
Oct 30 17:32:34.803214 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:34.803177    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fa2b418-d47e-4f0c-b0f5-36aa9dd14030-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d2d985-bw87p\" (UID: \"5fa2b418-d47e-4f0c-b0f5-36aa9dd14030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2d985-bw87p"
Oct 30 17:32:34.803439 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:34.803357    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp5tf\" (UniqueName: \"kubernetes.io/projected/5fa2b418-d47e-4f0c-b0f5-36aa9dd14030-kube-api-access-hp5tf\") pod \"maas-api-revoke-keys-e2e-ipp-a-d2d985-bw87p\" (UID: \"5fa2b418-d47e-4f0c-b0f5-36aa9dd14030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2d985-bw87p"
Oct 30 17:32:34.803439 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:34.803380    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fa2b418-d47e-4f0c-b0f5-36aa9dd14030-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d2d985-bw87p\" (UID: \"5fa2b418-d47e-4f0c-b0f5-36aa9dd14030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2d985-bw87p"
Oct 30 17:32:34.803988 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:34.803965    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fa2b418-d47e-4f0c-b0f5-36aa9dd14030-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d2d985-bw87p\" (UID: \"5fa2b418-d47e-4f0c-b0f5-36aa9dd14030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2d985-bw87p"
Oct 30 17:32:34.805808 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:34.805787    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fa2b418-d47e-4f0c-b0f5-36aa9dd14030-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d2d985-bw87p\" (UID: \"5fa2b418-d47e-4f0c-b0f5-36aa9dd14030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2d985-bw87p"
Oct 30 17:32:34.829725 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:34.829697    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp5tf\" (UniqueName: \"kubernetes.io/projected/5fa2b418-d47e-4f0c-b0f5-36aa9dd14030-kube-api-access-hp5tf\") pod \"maas-api-revoke-keys-e2e-ipp-a-d2d985-bw87p\" (UID: \"5fa2b418-d47e-4f0c-b0f5-36aa9dd14030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2d985-bw87p"
Oct 30 17:32:34.952369 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:34.952327    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2d985-bw87p"
Oct 30 17:32:35.305858 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:35.293910    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2d985-bw87p"]
Oct 30 17:32:35.560116 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:35.560041    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2d985-bw87p" event={"ID":"5fa2b418-d47e-4f0c-b0f5-36aa9dd14030","Type":"ContainerStarted","Data":"a32484ac19cb4ece83e67172ef607f588f88b5a2ad80afa7b7d5117bebd6d29c"}
Oct 30 17:32:36.568343 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:36.568305    4433 generic.go:356] "Generic (PLEG): container finished" podID="5fa2b418-d47e-4f0c-b0f5-36aa9dd14030" containerID="5dd6b88c1ab651473c99b6fefa002f2afbfbf778cda5c447f75d13684c7202b2" exitCode=0
Oct 30 17:32:36.568726 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:36.568378    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2d985-bw87p" event={"ID":"5fa2b418-d47e-4f0c-b0f5-36aa9dd14030","Type":"ContainerDied","Data":"5dd6b88c1ab651473c99b6fefa002f2afbfbf778cda5c447f75d13684c7202b2"}
Oct 30 17:32:37.756430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:37.756372    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2d985-bw87p"
Oct 30 17:32:37.861390 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:37.861299    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fa2b418-d47e-4f0c-b0f5-36aa9dd14030-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fa2b418-d47e-4f0c-b0f5-36aa9dd14030-maas-api-openshift-ca\") pod \"5fa2b418-d47e-4f0c-b0f5-36aa9dd14030\" (UID: \"5fa2b418-d47e-4f0c-b0f5-36aa9dd14030\") "
Oct 30 17:32:37.861597 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:37.861578    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fa2b418-d47e-4f0c-b0f5-36aa9dd14030-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fa2b418-d47e-4f0c-b0f5-36aa9dd14030-maas-api-xks-ca\") pod \"5fa2b418-d47e-4f0c-b0f5-36aa9dd14030\" (UID: \"5fa2b418-d47e-4f0c-b0f5-36aa9dd14030\") "
Oct 30 17:32:37.861683 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:37.861609    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fa2b418-d47e-4f0c-b0f5-36aa9dd14030-kube-api-access-hp5tf\" (UniqueName: \"kubernetes.io/projected/5fa2b418-d47e-4f0c-b0f5-36aa9dd14030-kube-api-access-hp5tf\") pod \"5fa2b418-d47e-4f0c-b0f5-36aa9dd14030\" (UID: \"5fa2b418-d47e-4f0c-b0f5-36aa9dd14030\") "
Oct 30 17:32:37.861780 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:37.861754    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fa2b418-d47e-4f0c-b0f5-36aa9dd14030-maas-api-openshift-ca" pod "5fa2b418-d47e-4f0c-b0f5-36aa9dd14030" (UID: "5fa2b418-d47e-4f0c-b0f5-36aa9dd14030"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:32:37.862195 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:37.862170    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fa2b418-d47e-4f0c-b0f5-36aa9dd14030-maas-api-openshift-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:32:37.863757 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:37.863730    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fa2b418-d47e-4f0c-b0f5-36aa9dd14030-maas-api-xks-ca" pod "5fa2b418-d47e-4f0c-b0f5-36aa9dd14030" (UID: "5fa2b418-d47e-4f0c-b0f5-36aa9dd14030"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:32:37.863848 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:37.863763    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fa2b418-d47e-4f0c-b0f5-36aa9dd14030-kube-api-access-hp5tf" pod "5fa2b418-d47e-4f0c-b0f5-36aa9dd14030" (UID: "5fa2b418-d47e-4f0c-b0f5-36aa9dd14030"). InnerVolumeSpecName "kube-api-access-hp5tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:32:37.964035 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:37.963996    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fa2b418-d47e-4f0c-b0f5-36aa9dd14030-maas-api-xks-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:32:37.964035 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:37.964029    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp5tf\" (UniqueName: \"kubernetes.io/projected/5fa2b418-d47e-4f0c-b0f5-36aa9dd14030-kube-api-access-hp5tf\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:32:38.583667 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:38.583633    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2d985-bw87p" event={"ID":"5fa2b418-d47e-4f0c-b0f5-36aa9dd14030","Type":"ContainerDied","Data":"a32484ac19cb4ece83e67172ef607f588f88b5a2ad80afa7b7d5117bebd6d29c"}
Oct 30 17:32:38.583667 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:38.583668    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2d985-bw87p"
Oct 30 17:32:38.583896 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:38.583716    4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a32484ac19cb4ece83e67172ef607f588f88b5a2ad80afa7b7d5117bebd6d29c"
Oct 30 17:32:54.889933 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:54.889833    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-84c5bf86f8-54rcv"]
Oct 30 17:32:55.687430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:55.677320    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd"]
Oct 30 17:32:55.687430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:55.677585    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd" podUID="1cd24c86-aae9-453b-a7a7-23296a87c420" containerName="maas-api" containerID="cri-o://ee907c4024aead38c212dcf43378fa065e6822c9a066adf40e9efb729aeac24a" gracePeriod=30
Oct 30 17:32:56.237325 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:56.237280    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:32:56.375484 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:56.373797    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd"
Oct 30 17:32:56.442802 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:56.435139    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cd24c86-aae9-453b-a7a7-23296a87c420-kube-api-access-hpqs9\" (UniqueName: \"kubernetes.io/projected/1cd24c86-aae9-453b-a7a7-23296a87c420-kube-api-access-hpqs9\") pod \"1cd24c86-aae9-453b-a7a7-23296a87c420\" (UID: \"1cd24c86-aae9-453b-a7a7-23296a87c420\") "
Oct 30 17:32:56.442802 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:56.435274    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cd24c86-aae9-453b-a7a7-23296a87c420-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd24c86-aae9-453b-a7a7-23296a87c420-openshift-service-ca\") pod \"1cd24c86-aae9-453b-a7a7-23296a87c420\" (UID: \"1cd24c86-aae9-453b-a7a7-23296a87c420\") "
Oct 30 17:32:56.442802 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:56.441777    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cd24c86-aae9-453b-a7a7-23296a87c420-openshift-service-ca" pod "1cd24c86-aae9-453b-a7a7-23296a87c420" (UID: "1cd24c86-aae9-453b-a7a7-23296a87c420"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:32:56.491421 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:56.491099    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cd24c86-aae9-453b-a7a7-23296a87c420-kube-api-access-hpqs9" pod "1cd24c86-aae9-453b-a7a7-23296a87c420" (UID: "1cd24c86-aae9-453b-a7a7-23296a87c420"). InnerVolumeSpecName "kube-api-access-hpqs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:32:56.550477 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:56.540175    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cd24c86-aae9-453b-a7a7-23296a87c420-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cd24c86-aae9-453b-a7a7-23296a87c420-maas-api-tls\") pod \"1cd24c86-aae9-453b-a7a7-23296a87c420\" (UID: \"1cd24c86-aae9-453b-a7a7-23296a87c420\") "
Oct 30 17:32:56.550477 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:56.540251    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cd24c86-aae9-453b-a7a7-23296a87c420-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cd24c86-aae9-453b-a7a7-23296a87c420-metrics-certs\") pod \"1cd24c86-aae9-453b-a7a7-23296a87c420\" (UID: \"1cd24c86-aae9-453b-a7a7-23296a87c420\") "
Oct 30 17:32:56.550477 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:56.540989    4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd24c86-aae9-453b-a7a7-23296a87c420-openshift-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:32:56.550477 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:56.541012    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpqs9\" (UniqueName: \"kubernetes.io/projected/1cd24c86-aae9-453b-a7a7-23296a87c420-kube-api-access-hpqs9\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:32:56.560469 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:56.559352    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cd24c86-aae9-453b-a7a7-23296a87c420-metrics-certs" pod "1cd24c86-aae9-453b-a7a7-23296a87c420" (UID: "1cd24c86-aae9-453b-a7a7-23296a87c420"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:32:56.568525 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:56.568485    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cd24c86-aae9-453b-a7a7-23296a87c420-maas-api-tls" pod "1cd24c86-aae9-453b-a7a7-23296a87c420" (UID: "1cd24c86-aae9-453b-a7a7-23296a87c420"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:32:56.642737 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:56.642652    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cd24c86-aae9-453b-a7a7-23296a87c420-maas-api-tls\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:32:56.642737 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:56.642681    4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cd24c86-aae9-453b-a7a7-23296a87c420-metrics-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:32:56.783382 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:56.783345    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-987dfc-b6gf5"]
Oct 30 17:32:56.788341 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:56.788301    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-987dfc-b6gf5"]
Oct 30 17:32:56.976648 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:56.976611    4433 generic.go:356] "Generic (PLEG): container finished" podID="1cd24c86-aae9-453b-a7a7-23296a87c420" containerID="ee907c4024aead38c212dcf43378fa065e6822c9a066adf40e9efb729aeac24a" exitCode=0
Oct 30 17:32:56.976831 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:56.976661    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd" event={"ID":"1cd24c86-aae9-453b-a7a7-23296a87c420","Type":"ContainerDied","Data":"ee907c4024aead38c212dcf43378fa065e6822c9a066adf40e9efb729aeac24a"}
Oct 30 17:32:56.976831 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:56.976731    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd" event={"ID":"1cd24c86-aae9-453b-a7a7-23296a87c420","Type":"ContainerDied","Data":"21f8d96fa819b5d1332880dbb0b3c56cb510cc18a64f3d1b48a3470d55bb5024"}
Oct 30 17:32:56.976831 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:56.976676    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd"
Oct 30 17:32:56.976831 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:56.976765    4433 scope.go:122] "RemoveContainer" containerID="ee907c4024aead38c212dcf43378fa065e6822c9a066adf40e9efb729aeac24a"
Oct 30 17:32:56.981345 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:32:56.981316    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:32:56.988667 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:56.988649    4433 scope.go:122] "RemoveContainer" containerID="ee907c4024aead38c212dcf43378fa065e6822c9a066adf40e9efb729aeac24a"
Oct 30 17:32:56.988912 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:32:56.988891    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee907c4024aead38c212dcf43378fa065e6822c9a066adf40e9efb729aeac24a\": container with ID starting with ee907c4024aead38c212dcf43378fa065e6822c9a066adf40e9efb729aeac24a not found: ID does not exist" containerID="ee907c4024aead38c212dcf43378fa065e6822c9a066adf40e9efb729aeac24a"
Oct 30 17:32:56.988959 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:56.988921    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee907c4024aead38c212dcf43378fa065e6822c9a066adf40e9efb729aeac24a"} err="failed to get container status \"ee907c4024aead38c212dcf43378fa065e6822c9a066adf40e9efb729aeac24a\": rpc error: code = NotFound desc = could not find container \"ee907c4024aead38c212dcf43378fa065e6822c9a066adf40e9efb729aeac24a\": container with ID starting with ee907c4024aead38c212dcf43378fa065e6822c9a066adf40e9efb729aeac24a not found: ID does not exist"
Oct 30 17:32:57.033828 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:57.033795    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd"]
Oct 30 17:32:57.038449 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:57.038424    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-987dfc-5667bcf54d-dmrgd"]
Oct 30 17:32:57.728426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:57.722255    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cd24c86-aae9-453b-a7a7-23296a87c420" path="/var/lib/kubelet/pods/1cd24c86-aae9-453b-a7a7-23296a87c420/volumes"
Oct 30 17:32:57.728426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:57.723039    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e2cce20-83b4-4883-8fd3-4dba0cb837ff" path="/var/lib/kubelet/pods/6e2cce20-83b4-4883-8fd3-4dba0cb837ff/volumes"
Oct 30 17:32:59.891430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:32:59.890474    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-84c5bf86f8-54rcv" podUID="68747a7e-891d-465e-9682-df5b696b50ff" containerName="payload-processing" containerID="cri-o://b1db6655054b803b730f18f43637ca32b68a5e0fa66dba8d43f00b388d2bf26c" gracePeriod=25
Oct 30 17:33:00.046654 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:00.046619    4433 generic.go:356] "Generic (PLEG): container finished" podID="68747a7e-891d-465e-9682-df5b696b50ff" containerID="b1db6655054b803b730f18f43637ca32b68a5e0fa66dba8d43f00b388d2bf26c" exitCode=0
Oct 30 17:33:00.046844 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:00.046692    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-84c5bf86f8-54rcv" event={"ID":"68747a7e-891d-465e-9682-df5b696b50ff","Type":"ContainerDied","Data":"b1db6655054b803b730f18f43637ca32b68a5e0fa66dba8d43f00b388d2bf26c"}
Oct 30 17:33:00.545493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:00.545456    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-84c5bf86f8-54rcv"
Oct 30 17:33:00.672430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:00.651065    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68747a7e-891d-465e-9682-df5b696b50ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68747a7e-891d-465e-9682-df5b696b50ff-plugins-config-volume\") pod \"68747a7e-891d-465e-9682-df5b696b50ff\" (UID: \"68747a7e-891d-465e-9682-df5b696b50ff\") "
Oct 30 17:33:00.693524 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:00.685741    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68747a7e-891d-465e-9682-df5b696b50ff-kube-api-access-94bvk\" (UniqueName: \"kubernetes.io/projected/68747a7e-891d-465e-9682-df5b696b50ff-kube-api-access-94bvk\") pod \"68747a7e-891d-465e-9682-df5b696b50ff\" (UID: \"68747a7e-891d-465e-9682-df5b696b50ff\") "
Oct 30 17:33:00.693524 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:00.665881    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68747a7e-891d-465e-9682-df5b696b50ff-plugins-config-volume" pod "68747a7e-891d-465e-9682-df5b696b50ff" (UID: "68747a7e-891d-465e-9682-df5b696b50ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:33:00.803427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:00.800229    4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68747a7e-891d-465e-9682-df5b696b50ff-plugins-config-volume\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:00.878431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:00.867631    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68747a7e-891d-465e-9682-df5b696b50ff-kube-api-access-94bvk" pod "68747a7e-891d-465e-9682-df5b696b50ff" (UID: "68747a7e-891d-465e-9682-df5b696b50ff"). InnerVolumeSpecName "kube-api-access-94bvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:33:00.993991 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:00.919244    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94bvk\" (UniqueName: \"kubernetes.io/projected/68747a7e-891d-465e-9682-df5b696b50ff-kube-api-access-94bvk\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:01.412486 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:01.387710    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-84c5bf86f8-54rcv" event={"ID":"68747a7e-891d-465e-9682-df5b696b50ff","Type":"ContainerDied","Data":"97495338e8b0816b78cd4aa149431152164691a923caf78b5f62e791a190c18c"}
Oct 30 17:33:01.412879 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:01.412849    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-84c5bf86f8-54rcv"
Oct 30 17:33:01.434827 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:01.434792    4433 scope.go:122] "RemoveContainer" containerID="b1db6655054b803b730f18f43637ca32b68a5e0fa66dba8d43f00b388d2bf26c"
Oct 30 17:33:01.609535 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:01.609486    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-84c5bf86f8-54rcv"]
Oct 30 17:33:01.632572 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:01.632537    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9e9ce7-84c5bf86f8-54rcv"]
Oct 30 17:33:01.799442 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:01.785900    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68747a7e-891d-465e-9682-df5b696b50ff" path="/var/lib/kubelet/pods/68747a7e-891d-465e-9682-df5b696b50ff/volumes"
Oct 30 17:33:04.601426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:04.594540    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"]
Oct 30 17:33:04.601426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:04.594819    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh" podUID="13bdf6f8-1211-4221-aa10-480d161dd3f8" containerName="istio-proxy" containerID="cri-o://aa2cafb4ac7b106ccddc2cc0ad79a94eab52e91d1799f5567885753fc46d52e1" gracePeriod=30
Oct 30 17:33:04.880488 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:04.879644    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-001a96-q7w8n"]
Oct 30 17:33:04.902430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:04.880699    4433 cpu_manager.go:414] "containerMap: removing container" podUID="68747a7e-891d-465e-9682-df5b696b50ff" containerName="payload-processing"
Oct 30 17:33:04.902430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:04.880725    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68747a7e-891d-465e-9682-df5b696b50ff" containerName="payload-processing"
Oct 30 17:33:04.902430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:04.880747    4433 cpu_manager.go:414] "containerMap: removing container" podUID="1cd24c86-aae9-453b-a7a7-23296a87c420" containerName="maas-api"
Oct 30 17:33:04.902430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:04.880756    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cd24c86-aae9-453b-a7a7-23296a87c420" containerName="maas-api"
Oct 30 17:33:04.902430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:04.880796    4433 cpu_manager.go:414] "containerMap: removing container" podUID="5fa2b418-d47e-4f0c-b0f5-36aa9dd14030" containerName="revoke-keys"
Oct 30 17:33:04.902430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:04.880804    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fa2b418-d47e-4f0c-b0f5-36aa9dd14030" containerName="revoke-keys"
Oct 30 17:33:04.902430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:04.880937    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fa2b418-d47e-4f0c-b0f5-36aa9dd14030" containerName="revoke-keys"
Oct 30 17:33:04.902430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:04.880958    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="68747a7e-891d-465e-9682-df5b696b50ff" containerName="payload-processing"
Oct 30 17:33:04.902430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:04.880968    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cd24c86-aae9-453b-a7a7-23296a87c420" containerName="maas-api"
Oct 30 17:33:04.928451 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:04.920087    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-001a96-q7w8n"
Oct 30 17:33:04.938571 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:04.929829    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-001a96-q7w8n"]
Oct 30 17:33:04.995785 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:04.995747    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:33:05.001834 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:05.000761    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\""
Oct 30 17:33:05.026621 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:05.026583    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36bbedfd-e691-423a-8535-e51effa187a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-001a96-q7w8n\" (UID: \"36bbedfd-e691-423a-8535-e51effa187a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-001a96-q7w8n"
Oct 30 17:33:05.040435 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:05.033853    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36bbedfd-e691-423a-8535-e51effa187a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-001a96-q7w8n\" (UID: \"36bbedfd-e691-423a-8535-e51effa187a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-001a96-q7w8n"
Oct 30 17:33:05.040803 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:05.040776    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spjq8\" (UniqueName: \"kubernetes.io/projected/36bbedfd-e691-423a-8535-e51effa187a5-kube-api-access-spjq8\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-001a96-q7w8n\" (UID: \"36bbedfd-e691-423a-8535-e51effa187a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-001a96-q7w8n"
Oct 30 17:33:05.143130 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:05.143010    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36bbedfd-e691-423a-8535-e51effa187a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-001a96-q7w8n\" (UID: \"36bbedfd-e691-423a-8535-e51effa187a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-001a96-q7w8n"
Oct 30 17:33:05.143682 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:05.143656    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spjq8\" (UniqueName: \"kubernetes.io/projected/36bbedfd-e691-423a-8535-e51effa187a5-kube-api-access-spjq8\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-001a96-q7w8n\" (UID: \"36bbedfd-e691-423a-8535-e51effa187a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-001a96-q7w8n"
Oct 30 17:33:05.144201 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:05.144179    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36bbedfd-e691-423a-8535-e51effa187a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-001a96-q7w8n\" (UID: \"36bbedfd-e691-423a-8535-e51effa187a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-001a96-q7w8n"
Oct 30 17:33:05.146037 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:05.146017    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36bbedfd-e691-423a-8535-e51effa187a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-001a96-q7w8n\" (UID: \"36bbedfd-e691-423a-8535-e51effa187a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-001a96-q7w8n"
Oct 30 17:33:05.146451 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:05.146433    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36bbedfd-e691-423a-8535-e51effa187a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-001a96-q7w8n\" (UID: \"36bbedfd-e691-423a-8535-e51effa187a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-001a96-q7w8n"
Oct 30 17:33:05.209428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:05.182202    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spjq8\" (UniqueName: \"kubernetes.io/projected/36bbedfd-e691-423a-8535-e51effa187a5-kube-api-access-spjq8\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-001a96-q7w8n\" (UID: \"36bbedfd-e691-423a-8535-e51effa187a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-001a96-q7w8n"
Oct 30 17:33:05.348171 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:05.348126    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-001a96-q7w8n"
Oct 30 17:33:06.026288 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:06.026260    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-001a96-q7w8n"]
Oct 30 17:33:06.028241 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:33:06.028208    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36bbedfd_e691_423a_8535_e51effa187a5.slice/crio-794a6715ae982c03b617f4ba0513a5d776b91ef1c23fa1d7e87076d5c6102fa7 WatchSource:0}: Error finding container 794a6715ae982c03b617f4ba0513a5d776b91ef1c23fa1d7e87076d5c6102fa7: Status 404 returned error can't find the container with id 794a6715ae982c03b617f4ba0513a5d776b91ef1c23fa1d7e87076d5c6102fa7
Oct 30 17:33:06.038599 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:06.038566    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:33:06.038711 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:06.038617    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh" podUID="13bdf6f8-1211-4221-aa10-480d161dd3f8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:33:06.724311 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:06.724269    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-001a96-q7w8n" event={"ID":"36bbedfd-e691-423a-8535-e51effa187a5","Type":"ContainerStarted","Data":"0c579ac46245675fb05cf732740f8bcd8b1afa705d5e4b4d06dd0cf9d977bafb"}
Oct 30 17:33:06.724540 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:06.724356    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-001a96-q7w8n" event={"ID":"36bbedfd-e691-423a-8535-e51effa187a5","Type":"ContainerStarted","Data":"794a6715ae982c03b617f4ba0513a5d776b91ef1c23fa1d7e87076d5c6102fa7"}
Oct 30 17:33:06.783172 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:06.783112    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-001a96-q7w8n" podStartSLOduration=2.7830971829999998 podStartE2EDuration="2.783097183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:06.781617989 +0000 UTC m=+2763.216520190" watchObservedRunningTime="2026-10-30 17:33:06.783097183 +0000 UTC m=+2763.217999416"
Oct 30 17:33:07.739772 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:07.739729    4433 generic.go:356] "Generic (PLEG): container finished" podID="36bbedfd-e691-423a-8535-e51effa187a5" containerID="0c579ac46245675fb05cf732740f8bcd8b1afa705d5e4b4d06dd0cf9d977bafb" exitCode=0
Oct 30 17:33:07.740301 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:07.739890    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-001a96-q7w8n" event={"ID":"36bbedfd-e691-423a-8535-e51effa187a5","Type":"ContainerDied","Data":"0c579ac46245675fb05cf732740f8bcd8b1afa705d5e4b4d06dd0cf9d977bafb"}
Oct 30 17:33:08.990602 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:08.990565    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-001a96-q7w8n"
Oct 30 17:33:09.021465 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:09.021370    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36bbedfd-e691-423a-8535-e51effa187a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36bbedfd-e691-423a-8535-e51effa187a5-maas-api-openshift-ca\") pod \"36bbedfd-e691-423a-8535-e51effa187a5\" (UID: \"36bbedfd-e691-423a-8535-e51effa187a5\") "
Oct 30 17:33:09.021625 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:09.021598    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36bbedfd-e691-423a-8535-e51effa187a5-kube-api-access-spjq8\" (UniqueName: \"kubernetes.io/projected/36bbedfd-e691-423a-8535-e51effa187a5-kube-api-access-spjq8\") pod \"36bbedfd-e691-423a-8535-e51effa187a5\" (UID: \"36bbedfd-e691-423a-8535-e51effa187a5\") "
Oct 30 17:33:09.021695 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:09.021641    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36bbedfd-e691-423a-8535-e51effa187a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36bbedfd-e691-423a-8535-e51effa187a5-maas-api-xks-ca\") pod \"36bbedfd-e691-423a-8535-e51effa187a5\" (UID: \"36bbedfd-e691-423a-8535-e51effa187a5\") "
Oct 30 17:33:09.022018 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:09.021994    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36bbedfd-e691-423a-8535-e51effa187a5-maas-api-openshift-ca" pod "36bbedfd-e691-423a-8535-e51effa187a5" (UID: "36bbedfd-e691-423a-8535-e51effa187a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:33:09.022233 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:09.022214    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36bbedfd-e691-423a-8535-e51effa187a5-maas-api-openshift-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:09.023937 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:09.023914    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36bbedfd-e691-423a-8535-e51effa187a5-kube-api-access-spjq8" pod "36bbedfd-e691-423a-8535-e51effa187a5" (UID: "36bbedfd-e691-423a-8535-e51effa187a5"). InnerVolumeSpecName "kube-api-access-spjq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:33:09.024307 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:09.024280    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36bbedfd-e691-423a-8535-e51effa187a5-maas-api-xks-ca" pod "36bbedfd-e691-423a-8535-e51effa187a5" (UID: "36bbedfd-e691-423a-8535-e51effa187a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:33:09.124435 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:09.124032    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spjq8\" (UniqueName: \"kubernetes.io/projected/36bbedfd-e691-423a-8535-e51effa187a5-kube-api-access-spjq8\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:09.124435 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:09.124073    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36bbedfd-e691-423a-8535-e51effa187a5-maas-api-xks-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:09.776419 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:09.776370    4433 generic.go:356] "Generic (PLEG): container finished" podID="13bdf6f8-1211-4221-aa10-480d161dd3f8" containerID="aa2cafb4ac7b106ccddc2cc0ad79a94eab52e91d1799f5567885753fc46d52e1" exitCode=0
Oct 30 17:33:09.776601 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:09.776520    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh" event={"ID":"13bdf6f8-1211-4221-aa10-480d161dd3f8","Type":"ContainerDied","Data":"aa2cafb4ac7b106ccddc2cc0ad79a94eab52e91d1799f5567885753fc46d52e1"}
Oct 30 17:33:09.776601 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:09.776579    4433 scope.go:122] "RemoveContainer" containerID="c11e45dd799b0879162a088605376060ff0d44770d15d155b568365d3a6d7fb7"
Oct 30 17:33:09.782155 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:09.779727    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-001a96-q7w8n" event={"ID":"36bbedfd-e691-423a-8535-e51effa187a5","Type":"ContainerDied","Data":"794a6715ae982c03b617f4ba0513a5d776b91ef1c23fa1d7e87076d5c6102fa7"}
Oct 30 17:33:09.782155 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:09.779783    4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="794a6715ae982c03b617f4ba0513a5d776b91ef1c23fa1d7e87076d5c6102fa7"
Oct 30 17:33:09.782155 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:09.779854    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-001a96-q7w8n"
Oct 30 17:33:10.023206 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.023165    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:33:10.065370 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.065329    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13bdf6f8-1211-4221-aa10-480d161dd3f8-kube-api-access-hz5mr\" (UniqueName: \"kubernetes.io/projected/13bdf6f8-1211-4221-aa10-480d161dd3f8-kube-api-access-hz5mr\") pod \"13bdf6f8-1211-4221-aa10-480d161dd3f8\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") "
Oct 30 17:33:10.065612 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.065503    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13bdf6f8-1211-4221-aa10-480d161dd3f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13bdf6f8-1211-4221-aa10-480d161dd3f8-workload-socket\") pod \"13bdf6f8-1211-4221-aa10-480d161dd3f8\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") "
Oct 30 17:33:10.065612 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.065549    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13bdf6f8-1211-4221-aa10-480d161dd3f8-istio-token\" (UniqueName: \"kubernetes.io/projected/13bdf6f8-1211-4221-aa10-480d161dd3f8-istio-token\") pod \"13bdf6f8-1211-4221-aa10-480d161dd3f8\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") "
Oct 30 17:33:10.065725 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.065690    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13bdf6f8-1211-4221-aa10-480d161dd3f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13bdf6f8-1211-4221-aa10-480d161dd3f8-istio-data\") pod \"13bdf6f8-1211-4221-aa10-480d161dd3f8\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") "
Oct 30 17:33:10.065856 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.065753    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13bdf6f8-1211-4221-aa10-480d161dd3f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13bdf6f8-1211-4221-aa10-480d161dd3f8-istio-envoy\") pod \"13bdf6f8-1211-4221-aa10-480d161dd3f8\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") "
Oct 30 17:33:10.065856 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.065807    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13bdf6f8-1211-4221-aa10-480d161dd3f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13bdf6f8-1211-4221-aa10-480d161dd3f8-istio-podinfo\") pod \"13bdf6f8-1211-4221-aa10-480d161dd3f8\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") "
Oct 30 17:33:10.065856 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.065850    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13bdf6f8-1211-4221-aa10-480d161dd3f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13bdf6f8-1211-4221-aa10-480d161dd3f8-istiod-ca-cert\") pod \"13bdf6f8-1211-4221-aa10-480d161dd3f8\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") "
Oct 30 17:33:10.066085 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.066064    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13bdf6f8-1211-4221-aa10-480d161dd3f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13bdf6f8-1211-4221-aa10-480d161dd3f8-credential-socket\") pod \"13bdf6f8-1211-4221-aa10-480d161dd3f8\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") "
Oct 30 17:33:10.066162 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.066146    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13bdf6f8-1211-4221-aa10-480d161dd3f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13bdf6f8-1211-4221-aa10-480d161dd3f8-workload-certs\") pod \"13bdf6f8-1211-4221-aa10-480d161dd3f8\" (UID: \"13bdf6f8-1211-4221-aa10-480d161dd3f8\") "
Oct 30 17:33:10.066673 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.066643    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13bdf6f8-1211-4221-aa10-480d161dd3f8-istio-data" pod "13bdf6f8-1211-4221-aa10-480d161dd3f8" (UID: "13bdf6f8-1211-4221-aa10-480d161dd3f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:33:10.066969 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.066943    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13bdf6f8-1211-4221-aa10-480d161dd3f8-workload-certs" pod "13bdf6f8-1211-4221-aa10-480d161dd3f8" (UID: "13bdf6f8-1211-4221-aa10-480d161dd3f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:33:10.067038 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.067025    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13bdf6f8-1211-4221-aa10-480d161dd3f8-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:10.068300 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.068270    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13bdf6f8-1211-4221-aa10-480d161dd3f8-istiod-ca-cert" pod "13bdf6f8-1211-4221-aa10-480d161dd3f8" (UID: "13bdf6f8-1211-4221-aa10-480d161dd3f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:33:10.068630 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.068600    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13bdf6f8-1211-4221-aa10-480d161dd3f8-credential-socket" pod "13bdf6f8-1211-4221-aa10-480d161dd3f8" (UID: "13bdf6f8-1211-4221-aa10-480d161dd3f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:33:10.068744 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.068711    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13bdf6f8-1211-4221-aa10-480d161dd3f8-workload-socket" pod "13bdf6f8-1211-4221-aa10-480d161dd3f8" (UID: "13bdf6f8-1211-4221-aa10-480d161dd3f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:33:10.071177 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.070867    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13bdf6f8-1211-4221-aa10-480d161dd3f8-kube-api-access-hz5mr" pod "13bdf6f8-1211-4221-aa10-480d161dd3f8" (UID: "13bdf6f8-1211-4221-aa10-480d161dd3f8"). InnerVolumeSpecName "kube-api-access-hz5mr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:33:10.071453 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.071384    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13bdf6f8-1211-4221-aa10-480d161dd3f8-istio-envoy" pod "13bdf6f8-1211-4221-aa10-480d161dd3f8" (UID: "13bdf6f8-1211-4221-aa10-480d161dd3f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:33:10.071887 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.071852    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13bdf6f8-1211-4221-aa10-480d161dd3f8-istio-token" pod "13bdf6f8-1211-4221-aa10-480d161dd3f8" (UID: "13bdf6f8-1211-4221-aa10-480d161dd3f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:33:10.073748 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.073717    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13bdf6f8-1211-4221-aa10-480d161dd3f8-istio-podinfo" pod "13bdf6f8-1211-4221-aa10-480d161dd3f8" (UID: "13bdf6f8-1211-4221-aa10-480d161dd3f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:33:10.168849 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.168804    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13bdf6f8-1211-4221-aa10-480d161dd3f8-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:10.168849 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.168845    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13bdf6f8-1211-4221-aa10-480d161dd3f8-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:10.169130 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.168862    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz5mr\" (UniqueName: \"kubernetes.io/projected/13bdf6f8-1211-4221-aa10-480d161dd3f8-kube-api-access-hz5mr\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:10.169130 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.168876    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13bdf6f8-1211-4221-aa10-480d161dd3f8-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:10.169130 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.168893    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13bdf6f8-1211-4221-aa10-480d161dd3f8-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:10.169130 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.168908    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13bdf6f8-1211-4221-aa10-480d161dd3f8-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:10.169130 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.168922    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13bdf6f8-1211-4221-aa10-480d161dd3f8-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:10.169130 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.168935    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13bdf6f8-1211-4221-aa10-480d161dd3f8-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:10.628604 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.628562    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"]
Oct 30 17:33:10.629841 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.629810    4433 cpu_manager.go:414] "containerMap: removing container" podUID="13bdf6f8-1211-4221-aa10-480d161dd3f8" containerName="istio-proxy"
Oct 30 17:33:10.629841 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.629838    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13bdf6f8-1211-4221-aa10-480d161dd3f8" containerName="istio-proxy"
Oct 30 17:33:10.630046 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.629857    4433 cpu_manager.go:414] "containerMap: removing container" podUID="13bdf6f8-1211-4221-aa10-480d161dd3f8" containerName="istio-proxy"
Oct 30 17:33:10.630046 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.629866    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13bdf6f8-1211-4221-aa10-480d161dd3f8" containerName="istio-proxy"
Oct 30 17:33:10.630046 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.629907    4433 cpu_manager.go:414] "containerMap: removing container" podUID="36bbedfd-e691-423a-8535-e51effa187a5" containerName="revoke-keys"
Oct 30 17:33:10.630046 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.629917    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36bbedfd-e691-423a-8535-e51effa187a5" containerName="revoke-keys"
Oct 30 17:33:10.630281 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.630082    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="36bbedfd-e691-423a-8535-e51effa187a5" containerName="revoke-keys"
Oct 30 17:33:10.630281 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.630108    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="13bdf6f8-1211-4221-aa10-480d161dd3f8" containerName="istio-proxy"
Oct 30 17:33:10.630281 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.630117    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="13bdf6f8-1211-4221-aa10-480d161dd3f8" containerName="istio-proxy"
Oct 30 17:33:10.669166 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.668502    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"
Oct 30 17:33:10.671687 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.670498    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"]
Oct 30 17:33:10.683933 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.683905    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-b87f3c-openshift-default-dockercfg-zqbjj\""
Oct 30 17:33:10.779582 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.779542    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r5s2\" (UniqueName: \"kubernetes.io/projected/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-kube-api-access-4r5s2\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"
Oct 30 17:33:10.779776 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.779651    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-istio-podinfo\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"
Oct 30 17:33:10.779848 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.779829    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-istio-data\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"
Oct 30 17:33:10.779950 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.779926    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-credential-socket\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"
Oct 30 17:33:10.780104 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.780021    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-workload-socket\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"
Oct 30 17:33:10.780197 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.780177    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-istiod-ca-cert\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"
Oct 30 17:33:10.780327 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.780264    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-workload-certs\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"
Oct 30 17:33:10.780393 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.780331    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-istio-envoy\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"
Oct 30 17:33:10.780393 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.780380    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-istio-token\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"
Oct 30 17:33:10.788340 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.788309    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh" event={"ID":"13bdf6f8-1211-4221-aa10-480d161dd3f8","Type":"ContainerDied","Data":"c958b2f4800da464ec41a65eafd07c9a555b090ebd7f2462839f39f2cb1c6fef"}
Oct 30 17:33:10.788481 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.788375    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"
Oct 30 17:33:10.788481 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.788383    4433 scope.go:122] "RemoveContainer" containerID="aa2cafb4ac7b106ccddc2cc0ad79a94eab52e91d1799f5567885753fc46d52e1"
Oct 30 17:33:10.843304 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.843223    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"]
Oct 30 17:33:10.849740 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.849710    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-vpchh"]
Oct 30 17:33:10.882907 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.882824    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r5s2\" (UniqueName: \"kubernetes.io/projected/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-kube-api-access-4r5s2\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"
Oct 30 17:33:10.883060 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.882903    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-istio-podinfo\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"
Oct 30 17:33:10.883109 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.883053    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-istio-data\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"
Oct 30 17:33:10.883159 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.883144    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-credential-socket\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"
Oct 30 17:33:10.883213 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.883194    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-workload-socket\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"
Oct 30 17:33:10.883310 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.883293    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-istiod-ca-cert\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"
Oct 30 17:33:10.883398 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.883347    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-workload-certs\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"
Oct 30 17:33:10.883398 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.883386    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-istio-envoy\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"
Oct 30 17:33:10.883529 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.883445    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-istio-token\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"
Oct 30 17:33:10.883529 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.883491    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-istio-data\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"
Oct 30 17:33:10.883640 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.883552    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-credential-socket\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"
Oct 30 17:33:10.883932 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.883909    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-workload-certs\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"
Oct 30 17:33:10.884163 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.884142    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-istiod-ca-cert\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"
Oct 30 17:33:10.884253 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.884239    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-workload-socket\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"
Oct 30 17:33:10.885734 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.885713    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-istio-envoy\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"
Oct 30 17:33:10.886203 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.886181    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-istio-podinfo\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"
Oct 30 17:33:10.905968 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.905916    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-istio-token\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"
Oct 30 17:33:10.906151 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:10.906002    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r5s2\" (UniqueName: \"kubernetes.io/projected/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-kube-api-access-4r5s2\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"
Oct 30 17:33:11.013844 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:11.013804    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"
Oct 30 17:33:12.679896 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:11.385733    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"]
Oct 30 17:33:12.679896 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:33:11.387700    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51cf9cfa_eed9_4472_ba73_d696ef7bbfcf.slice/crio-c360d7bcf148e344d96287397693b6d7b27aed10262d9da1cddcf555792fba7a WatchSource:0}: Error finding container c360d7bcf148e344d96287397693b6d7b27aed10262d9da1cddcf555792fba7a: Status 404 returned error can't find the container with id c360d7bcf148e344d96287397693b6d7b27aed10262d9da1cddcf555792fba7a
Oct 30 17:33:12.679896 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:11.390644    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:33:12.679896 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:11.390705    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:33:12.679896 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:11.390731    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:33:12.679896 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:11.698086    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13bdf6f8-1211-4221-aa10-480d161dd3f8" path="/var/lib/kubelet/pods/13bdf6f8-1211-4221-aa10-480d161dd3f8/volumes"
Oct 30 17:33:12.679896 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:11.799008    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz" event={"ID":"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf","Type":"ContainerStarted","Data":"9f7e96e18abec969a58085659e4f318835ad3d96e24ca6f34274e76819bc4059"}
Oct 30 17:33:12.679896 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:11.799063    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz" event={"ID":"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf","Type":"ContainerStarted","Data":"c360d7bcf148e344d96287397693b6d7b27aed10262d9da1cddcf555792fba7a"}
Oct 30 17:33:12.679896 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:11.845033    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz" podStartSLOduration=1.845018 podStartE2EDuration="1.845018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:11.842844013 +0000 UTC m=+2768.277746232" watchObservedRunningTime="2026-10-30 17:33:11.845018 +0000 UTC m=+2768.279920205"
Oct 30 17:33:12.679896 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:12.014676    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"
Oct 30 17:33:12.679896 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:12.016438    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 30 17:33:12.679896 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:12.016479    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz" podUID="51cf9cfa-eed9-4472-ba73-d696ef7bbfcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 30 17:33:12.811027 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:12.810996    4433 generic.go:356] "Generic (PLEG): container finished" podID="5fcae069-8f9f-47e5-8d3a-82c248cf3c31" containerID="0468bdc637faecdad7c1817dd916908c2917732fc4d55b05b4779401cb8f5afc" exitCode=0
Oct 30 17:33:13.003921 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:12.811085    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft" event={"ID":"5fcae069-8f9f-47e5-8d3a-82c248cf3c31","Type":"ContainerDied","Data":"0468bdc637faecdad7c1817dd916908c2917732fc4d55b05b4779401cb8f5afc"}
Oct 30 17:33:13.003921 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:12.811674    4433 scope.go:122] "RemoveContainer" containerID="0468bdc637faecdad7c1817dd916908c2917732fc4d55b05b4779401cb8f5afc"
Oct 30 17:33:13.003921 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:12.812814    4433 generic.go:356] "Generic (PLEG): container finished" podID="34119e6e-443a-40df-a54a-0f03b4e34553" containerID="db7028eeec58108cdd61659d39cc8be05b3e56e49b004f2268765a17af8eb4c2" exitCode=0
Oct 30 17:33:13.003921 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:12.812881    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg" event={"ID":"34119e6e-443a-40df-a54a-0f03b4e34553","Type":"ContainerDied","Data":"db7028eeec58108cdd61659d39cc8be05b3e56e49b004f2268765a17af8eb4c2"}
Oct 30 17:33:13.003921 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:12.813390    4433 scope.go:122] "RemoveContainer" containerID="db7028eeec58108cdd61659d39cc8be05b3e56e49b004f2268765a17af8eb4c2"
Oct 30 17:33:13.003921 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:12.814223    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:33:13.003921 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:12.814300    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:33:13.003921 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:12.814339    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:33:13.003921 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:12.815060    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:33:13.003921 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:12.815116    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:33:13.003921 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:12.815148    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:33:13.014675 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:13.014631    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 30 17:33:13.014815 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:13.014752    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz" podUID="51cf9cfa-eed9-4472-ba73-d696ef7bbfcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 30 17:33:13.821421 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:13.821369    4433 generic.go:356] "Generic (PLEG): container finished" podID="0cd29788-1dd4-4f3a-910e-0c180daed229" containerID="7c3a2a585b69006a59fa9b1fdcd7a0595f115e43433b23393e25ce2809c76c38" exitCode=0
Oct 30 17:33:13.821894 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:13.821437    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch" event={"ID":"0cd29788-1dd4-4f3a-910e-0c180daed229","Type":"ContainerDied","Data":"7c3a2a585b69006a59fa9b1fdcd7a0595f115e43433b23393e25ce2809c76c38"}
Oct 30 17:33:13.821994 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:13.821975    4433 scope.go:122] "RemoveContainer" containerID="7c3a2a585b69006a59fa9b1fdcd7a0595f115e43433b23393e25ce2809c76c38"
Oct 30 17:33:13.824031 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:13.824008    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft" event={"ID":"5fcae069-8f9f-47e5-8d3a-82c248cf3c31","Type":"ContainerStarted","Data":"887486d0abbe40860e456f4aafa560edc08e7ebb1b23608ee565746b02f71998"}
Oct 30 17:33:13.824222 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:13.824179    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:33:13.824349 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:13.824261    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:33:13.824349 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:13.824301    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:33:13.826901 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:13.826869    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg" event={"ID":"34119e6e-443a-40df-a54a-0f03b4e34553","Type":"ContainerStarted","Data":"d42a603f0a2c6d073f6d62a247feb87294dc3ddd4933800e8df490589a1d8c1b"}
Oct 30 17:33:13.883128 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:13.882977    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:33:13.886125 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:13.885774    4433 patch_prober.go:29] interesting pod/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 17:33:13.886125 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:13.885828    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft" podUID="5fcae069-8f9f-47e5-8d3a-82c248cf3c31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 17:33:14.013968 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:14.013933    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:33:14.014197 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:14.014173    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 30 17:33:14.014274 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:14.014223    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz" podUID="51cf9cfa-eed9-4472-ba73-d696ef7bbfcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 30 17:33:14.014377 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:14.014360    4433 patch_prober.go:29] interesting pod/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 17:33:14.014513 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:14.014391    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg" podUID="34119e6e-443a-40df-a54a-0f03b4e34553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 17:33:14.837910 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:14.837870    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch" event={"ID":"0cd29788-1dd4-4f3a-910e-0c180daed229","Type":"ContainerStarted","Data":"87903d0308a7de56c43842895dc53610793ff799db9e95c55330d904c978c715"}
Oct 30 17:33:14.881557 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:14.881514    4433 patch_prober.go:29] interesting pod/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 17:33:14.882038 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:14.882003    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft" podUID="5fcae069-8f9f-47e5-8d3a-82c248cf3c31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 17:33:15.026926 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:15.026882    4433 patch_prober.go:29] interesting pod/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 17:33:15.027211 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:15.027177    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg" podUID="34119e6e-443a-40df-a54a-0f03b4e34553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 17:33:15.028072 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:15.028045    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 30 17:33:15.028248 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:15.028227    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz" podUID="51cf9cfa-eed9-4472-ba73-d696ef7bbfcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 30 17:33:15.159525 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:15.158100    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:33:15.159525 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:15.158283    4433 patch_prober.go:29] interesting pod/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 17:33:15.159525 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:15.158329    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch" podUID="0cd29788-1dd4-4f3a-910e-0c180daed229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 17:33:15.587249 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:15.587210    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41cb25-rbqcp"]
Oct 30 17:33:15.657625 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:15.653428    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41cb25-rbqcp"
Oct 30 17:33:15.800874 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:15.800831    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\""
Oct 30 17:33:15.809240 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:15.809064    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:33:15.863890 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:15.851523    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41cb25-rbqcp"]
Oct 30 17:33:15.883636 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:15.883601    4433 patch_prober.go:29] interesting pod/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 17:33:15.883962 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:15.883913    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft" podUID="5fcae069-8f9f-47e5-8d3a-82c248cf3c31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 17:33:15.896451 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:15.893961    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506303c1-2a19-4487-a657-c10901f0c16f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-41cb25-rbqcp\" (UID: \"506303c1-2a19-4487-a657-c10901f0c16f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41cb25-rbqcp"
Oct 30 17:33:15.954880 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:15.929457    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8r6k\" (UniqueName: \"kubernetes.io/projected/506303c1-2a19-4487-a657-c10901f0c16f-kube-api-access-t8r6k\") pod \"maas-api-revoke-keys-e2e-ipp-a-41cb25-rbqcp\" (UID: \"506303c1-2a19-4487-a657-c10901f0c16f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41cb25-rbqcp"
Oct 30 17:33:15.954880 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:15.929586    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506303c1-2a19-4487-a657-c10901f0c16f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-41cb25-rbqcp\" (UID: \"506303c1-2a19-4487-a657-c10901f0c16f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41cb25-rbqcp"
Oct 30 17:33:16.015520 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:16.015041    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 30 17:33:16.015520 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:16.015102    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz" podUID="51cf9cfa-eed9-4472-ba73-d696ef7bbfcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 30 17:33:16.016027 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:16.015923    4433 patch_prober.go:29] interesting pod/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 17:33:16.016027 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:16.015969    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg" podUID="34119e6e-443a-40df-a54a-0f03b4e34553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 17:33:16.042621 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:16.031392    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8r6k\" (UniqueName: \"kubernetes.io/projected/506303c1-2a19-4487-a657-c10901f0c16f-kube-api-access-t8r6k\") pod \"maas-api-revoke-keys-e2e-ipp-a-41cb25-rbqcp\" (UID: \"506303c1-2a19-4487-a657-c10901f0c16f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41cb25-rbqcp"
Oct 30 17:33:16.042621 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:16.031523    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506303c1-2a19-4487-a657-c10901f0c16f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-41cb25-rbqcp\" (UID: \"506303c1-2a19-4487-a657-c10901f0c16f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41cb25-rbqcp"
Oct 30 17:33:16.042621 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:16.031712    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506303c1-2a19-4487-a657-c10901f0c16f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-41cb25-rbqcp\" (UID: \"506303c1-2a19-4487-a657-c10901f0c16f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41cb25-rbqcp"
Oct 30 17:33:16.042621 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:16.032808    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506303c1-2a19-4487-a657-c10901f0c16f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-41cb25-rbqcp\" (UID: \"506303c1-2a19-4487-a657-c10901f0c16f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41cb25-rbqcp"
Oct 30 17:33:16.063620 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:16.062568    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 17:33:16.097555 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:16.097520    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8r6k\" (UniqueName: \"kubernetes.io/projected/506303c1-2a19-4487-a657-c10901f0c16f-kube-api-access-t8r6k\") pod \"maas-api-revoke-keys-e2e-ipp-a-41cb25-rbqcp\" (UID: \"506303c1-2a19-4487-a657-c10901f0c16f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41cb25-rbqcp"
Oct 30 17:33:16.112687 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:16.111614    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506303c1-2a19-4487-a657-c10901f0c16f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-41cb25-rbqcp\" (UID: \"506303c1-2a19-4487-a657-c10901f0c16f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41cb25-rbqcp"
Oct 30 17:33:16.112687 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:16.111886    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41cb25-rbqcp"
Oct 30 17:33:16.165044 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:16.164255    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq" podUID="ba03c7a7-48e4-4051-91d4-9a030790fa8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.168:8000/ready\": context deadline exceeded"
Oct 30 17:33:16.283636 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:16.269569    4433 patch_prober.go:29] interesting pod/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 17:33:16.283636 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:16.269629    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch" podUID="0cd29788-1dd4-4f3a-910e-0c180daed229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 17:33:16.367627 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:16.352621    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:33:16.448679 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:16.446030    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:16.473448 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:16.454538    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:16.612429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:16.598559    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 17:33:16.699492 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:16.699350    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:16.723429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:16.720474    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:33:16.896566 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:16.896508    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" podUID="458eba2e-eb03-452a-bcf4-a5e91da83acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:16.913349 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:16.911518    4433 patch_prober.go:29] interesting pod/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 17:33:16.913349 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:16.911575    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft" podUID="5fcae069-8f9f-47e5-8d3a-82c248cf3c31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 17:33:17.053470 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:17.052799    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:17.115781 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:17.094593    4433 patch_prober.go:29] interesting pod/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 17:33:17.115781 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:17.094664    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg" podUID="34119e6e-443a-40df-a54a-0f03b4e34553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 17:33:17.115781 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:17.096724    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 30 17:33:17.115781 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:17.096766    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz" podUID="51cf9cfa-eed9-4472-ba73-d696ef7bbfcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 30 17:33:17.115781 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:17.097082    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 17:33:17.115781 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:17.097377    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 17:33:17.115781 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:17.097710    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:33:17.279479 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:17.267681    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq" podUID="ba03c7a7-48e4-4051-91d4-9a030790fa8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.168:8000/ready\": context deadline exceeded"
Oct 30 17:33:17.303426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:17.283341    4433 patch_prober.go:29] interesting pod/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 17:33:17.303426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:17.300916    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch" podUID="0cd29788-1dd4-4f3a-910e-0c180daed229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 17:33:17.303426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:17.301024    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq"
Oct 30 17:33:17.455510 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:17.455466    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:17.457260 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:17.457231    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 17:33:17.748428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:17.741133    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 17:33:17.777432 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:17.764385    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm" podUID="bf43bf3d-f126-40a1-a24f-10214a85ceb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 30 17:33:17.777432 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:17.772900    4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:17.777432 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:17.776264    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:17.777432 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:17.776722    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:33:17.802429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:17.778132    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 17:33:17.994429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:17.947534    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41cb25-rbqcp"]
Oct 30 17:33:17.994429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:17.948575    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" podUID="458eba2e-eb03-452a-bcf4-a5e91da83acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 17:33:17.994429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:17.959624    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:33:17.995123 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:17.920648    4433 patch_prober.go:29] interesting pod/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 17:33:18.025681 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:18.025575    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft" podUID="5fcae069-8f9f-47e5-8d3a-82c248cf3c31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 17:33:18.039874 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:18.029553    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 30 17:33:18.040165 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:18.040129    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz" podUID="51cf9cfa-eed9-4472-ba73-d696ef7bbfcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 30 17:33:18.048846 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:18.029619    4433 patch_prober.go:29] interesting pod/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 17:33:18.048846 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:18.040480    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg" podUID="34119e6e-443a-40df-a54a-0f03b4e34553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 17:33:18.076132 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:18.076087    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 17:33:18.077491 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:18.077450    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 17:33:18.139480 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:18.098269    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 17:33:18.139480 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:18.111601    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:18.139480 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:18.115539    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 17:33:18.184881 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:18.184844    4433 patch_prober.go:29] interesting pod/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 17:33:18.185153 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:18.185124    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch" podUID="0cd29788-1dd4-4f3a-910e-0c180daed229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 17:33:18.493969 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:18.493911    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-001a96-q7w8n"]
Oct 30 17:33:18.494503 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:18.494479    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 17:33:18.522291 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:18.522245    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq" podUID="ba03c7a7-48e4-4051-91d4-9a030790fa8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.168:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:18.634491 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:18.550921    4433 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq" podUID="ba03c7a7-48e4-4051-91d4-9a030790fa8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.168:8000/health\": context deadline exceeded"
Oct 30 17:33:18.634491 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:18.586526    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:18.903605 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:18.900845    4433 patch_prober.go:29] interesting pod/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 17:33:18.903605 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:18.900936    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft" podUID="5fcae069-8f9f-47e5-8d3a-82c248cf3c31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 17:33:18.960430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:18.939322    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:19.186976 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:18.939999    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 30 17:33:19.186976 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:18.940033    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm" podUID="bf43bf3d-f126-40a1-a24f-10214a85ceb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:19.186976 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:18.942682    4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded"
Oct 30 17:33:19.186976 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:18.962805    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" podUID="458eba2e-eb03-452a-bcf4-a5e91da83acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:19.186976 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:19.015285    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-001a96-q7w8n"]
Oct 30 17:33:19.186976 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:19.174078    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41cb25-rbqcp" event={"ID":"506303c1-2a19-4487-a657-c10901f0c16f","Type":"ContainerStarted","Data":"dada338e3b81a62101e27c245ccf7b63f519a99a072c9efe7d5f9f3a6c18147a"}
Oct 30 17:33:19.186976 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:19.174161    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 17:33:19.186976 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:19.174204    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 17:33:19.186976 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:19.128587    4433 patch_prober.go:29] interesting pod/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 17:33:19.186976 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:19.174573    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg" podUID="34119e6e-443a-40df-a54a-0f03b4e34553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 17:33:19.186976 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:19.145104    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:19.186976 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:19.145857    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 30 17:33:19.186976 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:19.175105    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz" podUID="51cf9cfa-eed9-4472-ba73-d696ef7bbfcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 30 17:33:19.186976 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:19.168985    4433 patch_prober.go:29] interesting pod/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 17:33:19.186976 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:19.175149    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch" podUID="0cd29788-1dd4-4f3a-910e-0c180daed229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 17:33:19.484484 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:19.483734    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:19.596554 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:19.596500    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:19.667482 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:19.625919    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq" podUID="ba03c7a7-48e4-4051-91d4-9a030790fa8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.168:8000/ready\": context deadline exceeded"
Oct 30 17:33:19.668953 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:19.667832    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:19.896450 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:19.880325    4433 patch_prober.go:29] interesting pod/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 17:33:19.896450 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:19.880384    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft" podUID="5fcae069-8f9f-47e5-8d3a-82c248cf3c31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 17:33:19.988335 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:19.988292    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36bbedfd-e691-423a-8535-e51effa187a5" path="/var/lib/kubelet/pods/36bbedfd-e691-423a-8535-e51effa187a5/volumes"
Oct 30 17:33:20.002428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:19.989107    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:33:20.013768 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:20.013733    4433 patch_prober.go:29] interesting pod/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 17:33:20.014822 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:20.014793    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg" podUID="34119e6e-443a-40df-a54a-0f03b4e34553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 17:33:20.062003 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:20.061970    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:33:20.138840 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:20.138738    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 17:33:20.213436 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:20.159669    4433 patch_prober.go:29] interesting pod/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 17:33:20.213436 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:20.159741    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch" podUID="0cd29788-1dd4-4f3a-910e-0c180daed229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 17:33:20.213436 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:20.194782    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 17:33:20.454488 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:20.326083    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm" podUID="bf43bf3d-f126-40a1-a24f-10214a85ceb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:20.485033 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:20.352692    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 30 17:33:20.512302 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:20.370871    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:33:20.514126 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:20.395380    4433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:20.514323 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:20.490199    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 17:33:20.523439 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:20.515193    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 17:33:20.567370 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:20.567338    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"
Oct 30 17:33:20.587017 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:20.579078    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"
Oct 30 17:33:20.597257 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:20.597213    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 17:33:20.598581 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:20.598558    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:33:20.647551 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:20.613994    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"
Oct 30 17:33:20.680766 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:20.669180    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq" podUID="ba03c7a7-48e4-4051-91d4-9a030790fa8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.168:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:20.680766 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:20.669781    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:20.995428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:20.994004    4433 patch_prober.go:29] interesting pod/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 17:33:20.995428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:20.994066    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft" podUID="5fcae069-8f9f-47e5-8d3a-82c248cf3c31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 17:33:21.104763 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:21.070884    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:21.104763 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:21.088151    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41cb25-rbqcp" event={"ID":"506303c1-2a19-4487-a657-c10901f0c16f","Type":"ContainerStarted","Data":"f217cf4f0f265e1956b307ea0ac3937ab98cdac725754e158a879779853527fe"}
Oct 30 17:33:21.139464 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:21.139379    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:21.140180 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:21.140153    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 17:33:21.173430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:21.172097    4433 patch_prober.go:29] interesting pod/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 17:33:21.173430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:21.172161    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch" podUID="0cd29788-1dd4-4f3a-910e-0c180daed229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 17:33:21.188432 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:21.183390    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 17:33:21.193090 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:21.193058    4433 patch_prober.go:29] interesting pod/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 17:33:21.193322 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:21.193296    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg" podUID="34119e6e-443a-40df-a54a-0f03b4e34553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 17:33:21.228432 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:21.198637    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:21.244358 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:21.244204    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 17:33:21.432498 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:21.421904    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:33:21.623467 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:21.611300    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41cb25-rbqcp" podStartSLOduration=6.611277319 podStartE2EDuration="6.611277319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:21.231647089 +0000 UTC m=+2777.666549298" watchObservedRunningTime="2026-10-30 17:33:21.611277319 +0000 UTC m=+2778.046179526"
Oct 30 17:33:21.646921 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:21.646869    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm" podUID="bf43bf3d-f126-40a1-a24f-10214a85ceb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 30 17:33:21.679657 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:21.679581    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq" podUID="ba03c7a7-48e4-4051-91d4-9a030790fa8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.168:8000/ready\": context deadline exceeded"
Oct 30 17:33:21.711695 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:21.711587    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:21.746166 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:21.746123    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:21.816079 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:21.811483    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 17:33:21.816079 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:21.811542    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:33:21.816079 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:21.811573    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 17:33:21.832696 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:21.832664    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq"
Oct 30 17:33:21.833494 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:21.833470    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 17:33:21.935493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:21.931709    4433 patch_prober.go:29] interesting pod/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 17:33:21.935493 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:21.931786    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft" podUID="5fcae069-8f9f-47e5-8d3a-82c248cf3c31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 17:33:22.112359 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:22.112259    4433 generic.go:356] "Generic (PLEG): container finished" podID="506303c1-2a19-4487-a657-c10901f0c16f" containerID="f217cf4f0f265e1956b307ea0ac3937ab98cdac725754e158a879779853527fe" exitCode=0
Oct 30 17:33:22.112808 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:22.112704    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41cb25-rbqcp" event={"ID":"506303c1-2a19-4487-a657-c10901f0c16f","Type":"ContainerDied","Data":"f217cf4f0f265e1956b307ea0ac3937ab98cdac725754e158a879779853527fe"}
Oct 30 17:33:22.136447 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:22.134954    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:33:22.139172 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:22.139146    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:33:22.177481 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:22.154792    4433 patch_prober.go:29] interesting pod/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 17:33:22.177481 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:22.154850    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch" podUID="0cd29788-1dd4-4f3a-910e-0c180daed229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 17:33:22.884136 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:22.884101    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:33:22.885322 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:22.885300    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:33:23.201992 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:23.198363    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:33:23.203203 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:23.202703    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:33:23.305912 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:23.305871    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41cb25-rbqcp"
Oct 30 17:33:23.325445 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:23.325384    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/506303c1-2a19-4487-a657-c10901f0c16f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506303c1-2a19-4487-a657-c10901f0c16f-maas-api-openshift-ca\") pod \"506303c1-2a19-4487-a657-c10901f0c16f\" (UID: \"506303c1-2a19-4487-a657-c10901f0c16f\") "
Oct 30 17:33:23.325730 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:23.325711    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/506303c1-2a19-4487-a657-c10901f0c16f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506303c1-2a19-4487-a657-c10901f0c16f-maas-api-xks-ca\") pod \"506303c1-2a19-4487-a657-c10901f0c16f\" (UID: \"506303c1-2a19-4487-a657-c10901f0c16f\") "
Oct 30 17:33:23.325816 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:23.325746    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/506303c1-2a19-4487-a657-c10901f0c16f-kube-api-access-t8r6k\" (UniqueName: \"kubernetes.io/projected/506303c1-2a19-4487-a657-c10901f0c16f-kube-api-access-t8r6k\") pod \"506303c1-2a19-4487-a657-c10901f0c16f\" (UID: \"506303c1-2a19-4487-a657-c10901f0c16f\") "
Oct 30 17:33:23.325816 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:23.325799    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/506303c1-2a19-4487-a657-c10901f0c16f-maas-api-openshift-ca" pod "506303c1-2a19-4487-a657-c10901f0c16f" (UID: "506303c1-2a19-4487-a657-c10901f0c16f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:33:23.329032 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:23.328934    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506303c1-2a19-4487-a657-c10901f0c16f-maas-api-openshift-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:23.329032 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:23.328964    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/506303c1-2a19-4487-a657-c10901f0c16f-maas-api-xks-ca" pod "506303c1-2a19-4487-a657-c10901f0c16f" (UID: "506303c1-2a19-4487-a657-c10901f0c16f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:33:23.335275 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:23.335247    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/506303c1-2a19-4487-a657-c10901f0c16f-kube-api-access-t8r6k" pod "506303c1-2a19-4487-a657-c10901f0c16f" (UID: "506303c1-2a19-4487-a657-c10901f0c16f"). InnerVolumeSpecName "kube-api-access-t8r6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:33:23.430047 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:23.430014    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506303c1-2a19-4487-a657-c10901f0c16f-maas-api-xks-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:23.430047 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:23.430046    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8r6k\" (UniqueName: \"kubernetes.io/projected/506303c1-2a19-4487-a657-c10901f0c16f-kube-api-access-t8r6k\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:24.127545 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:24.127457    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41cb25-rbqcp" event={"ID":"506303c1-2a19-4487-a657-c10901f0c16f","Type":"ContainerDied","Data":"dada338e3b81a62101e27c245ccf7b63f519a99a072c9efe7d5f9f3a6c18147a"}
Oct 30 17:33:24.127545 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:24.127515    4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dada338e3b81a62101e27c245ccf7b63f519a99a072c9efe7d5f9f3a6c18147a"
Oct 30 17:33:24.127545 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:24.127470    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41cb25-rbqcp"
Oct 30 17:33:25.529033 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:25.528982    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"]
Oct 30 17:33:25.529472 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:25.529295    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q" podUID="6906457b-d282-4603-8e72-edd173a4afa7" containerName="istio-proxy" containerID="cri-o://549a6c08fedba5e518e75ca00986b095623032ba88e99849abc50fc4d6ff2aaa" gracePeriod=30
Oct 30 17:33:26.233786 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:26.233752    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7b9df-2lphz"]
Oct 30 17:33:26.234446 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:26.234429    4433 cpu_manager.go:414] "containerMap: removing container" podUID="506303c1-2a19-4487-a657-c10901f0c16f" containerName="revoke-keys"
Oct 30 17:33:26.234491 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:26.234448    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="506303c1-2a19-4487-a657-c10901f0c16f" containerName="revoke-keys"
Oct 30 17:33:26.234581 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:26.234572    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="506303c1-2a19-4487-a657-c10901f0c16f" containerName="revoke-keys"
Oct 30 17:33:26.238353 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:26.238334    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7b9df-2lphz"
Oct 30 17:33:26.258197 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:26.258145    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7b9df-2lphz"]
Oct 30 17:33:26.304505 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:26.304467    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\""
Oct 30 17:33:26.304505 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:26.304469    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:33:26.334148 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:26.334110    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lb8g\" (UniqueName: \"kubernetes.io/projected/3199ec19-a07d-4225-a9d8-1926f089ae6b-kube-api-access-7lb8g\") pod \"maas-api-revoke-keys-e2e-worker-w4-c7b9df-2lphz\" (UID: \"3199ec19-a07d-4225-a9d8-1926f089ae6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7b9df-2lphz"
Oct 30 17:33:26.334301 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:26.334265    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3199ec19-a07d-4225-a9d8-1926f089ae6b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c7b9df-2lphz\" (UID: \"3199ec19-a07d-4225-a9d8-1926f089ae6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7b9df-2lphz"
Oct 30 17:33:26.334356 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:26.334337    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3199ec19-a07d-4225-a9d8-1926f089ae6b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c7b9df-2lphz\" (UID: \"3199ec19-a07d-4225-a9d8-1926f089ae6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7b9df-2lphz"
Oct 30 17:33:26.436037 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:26.436004    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3199ec19-a07d-4225-a9d8-1926f089ae6b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c7b9df-2lphz\" (UID: \"3199ec19-a07d-4225-a9d8-1926f089ae6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7b9df-2lphz"
Oct 30 17:33:26.436226 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:26.436079    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3199ec19-a07d-4225-a9d8-1926f089ae6b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c7b9df-2lphz\" (UID: \"3199ec19-a07d-4225-a9d8-1926f089ae6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7b9df-2lphz"
Oct 30 17:33:26.436314 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:26.436295    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lb8g\" (UniqueName: \"kubernetes.io/projected/3199ec19-a07d-4225-a9d8-1926f089ae6b-kube-api-access-7lb8g\") pod \"maas-api-revoke-keys-e2e-worker-w4-c7b9df-2lphz\" (UID: \"3199ec19-a07d-4225-a9d8-1926f089ae6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7b9df-2lphz"
Oct 30 17:33:26.436802 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:26.436778    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3199ec19-a07d-4225-a9d8-1926f089ae6b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c7b9df-2lphz\" (UID: \"3199ec19-a07d-4225-a9d8-1926f089ae6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7b9df-2lphz"
Oct 30 17:33:26.438729 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:26.438703    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3199ec19-a07d-4225-a9d8-1926f089ae6b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c7b9df-2lphz\" (UID: \"3199ec19-a07d-4225-a9d8-1926f089ae6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7b9df-2lphz"
Oct 30 17:33:26.460214 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:26.460181    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lb8g\" (UniqueName: \"kubernetes.io/projected/3199ec19-a07d-4225-a9d8-1926f089ae6b-kube-api-access-7lb8g\") pod \"maas-api-revoke-keys-e2e-worker-w4-c7b9df-2lphz\" (UID: \"3199ec19-a07d-4225-a9d8-1926f089ae6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7b9df-2lphz"
Oct 30 17:33:26.553380 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:26.553288    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7b9df-2lphz"
Oct 30 17:33:26.865968 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:26.865937    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7b9df-2lphz"]
Oct 30 17:33:26.867215 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:33:26.867184    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3199ec19_a07d_4225_a9d8_1926f089ae6b.slice/crio-809075724f6936f2254e24ac96769f800dcdc493b56cbc9cbb620f5abf07c88c WatchSource:0}: Error finding container 809075724f6936f2254e24ac96769f800dcdc493b56cbc9cbb620f5abf07c88c: Status 404 returned error can't find the container with id 809075724f6936f2254e24ac96769f800dcdc493b56cbc9cbb620f5abf07c88c
Oct 30 17:33:26.979292 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:33:26.979253    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:33:27.149637 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:27.149549    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7b9df-2lphz" event={"ID":"3199ec19-a07d-4225-a9d8-1926f089ae6b","Type":"ContainerStarted","Data":"809075724f6936f2254e24ac96769f800dcdc493b56cbc9cbb620f5abf07c88c"}
Oct 30 17:33:28.157955 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:28.157868    4433 generic.go:356] "Generic (PLEG): container finished" podID="3199ec19-a07d-4225-a9d8-1926f089ae6b" containerID="4efbdf4e1551f7165b9fc06cc98d70cb25b8da019beca492355ad64c42591e69" exitCode=0
Oct 30 17:33:28.158348 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:28.157953    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7b9df-2lphz" event={"ID":"3199ec19-a07d-4225-a9d8-1926f089ae6b","Type":"ContainerDied","Data":"4efbdf4e1551f7165b9fc06cc98d70cb25b8da019beca492355ad64c42591e69"}
Oct 30 17:33:29.396851 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:29.396822    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7b9df-2lphz"
Oct 30 17:33:29.427607 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:29.427578    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3199ec19-a07d-4225-a9d8-1926f089ae6b-kube-api-access-7lb8g\" (UniqueName: \"kubernetes.io/projected/3199ec19-a07d-4225-a9d8-1926f089ae6b-kube-api-access-7lb8g\") pod \"3199ec19-a07d-4225-a9d8-1926f089ae6b\" (UID: \"3199ec19-a07d-4225-a9d8-1926f089ae6b\") "
Oct 30 17:33:29.427792 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:29.427709    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3199ec19-a07d-4225-a9d8-1926f089ae6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3199ec19-a07d-4225-a9d8-1926f089ae6b-maas-api-openshift-ca\") pod \"3199ec19-a07d-4225-a9d8-1926f089ae6b\" (UID: \"3199ec19-a07d-4225-a9d8-1926f089ae6b\") "
Oct 30 17:33:29.427871 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:29.427822    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3199ec19-a07d-4225-a9d8-1926f089ae6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3199ec19-a07d-4225-a9d8-1926f089ae6b-maas-api-xks-ca\") pod \"3199ec19-a07d-4225-a9d8-1926f089ae6b\" (UID: \"3199ec19-a07d-4225-a9d8-1926f089ae6b\") "
Oct 30 17:33:29.428161 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:29.428133    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3199ec19-a07d-4225-a9d8-1926f089ae6b-maas-api-openshift-ca" pod "3199ec19-a07d-4225-a9d8-1926f089ae6b" (UID: "3199ec19-a07d-4225-a9d8-1926f089ae6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:33:29.428309 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:29.428292    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3199ec19-a07d-4225-a9d8-1926f089ae6b-maas-api-openshift-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:29.429958 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:29.429936    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3199ec19-a07d-4225-a9d8-1926f089ae6b-maas-api-xks-ca" pod "3199ec19-a07d-4225-a9d8-1926f089ae6b" (UID: "3199ec19-a07d-4225-a9d8-1926f089ae6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:33:29.430237 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:29.430214    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3199ec19-a07d-4225-a9d8-1926f089ae6b-kube-api-access-7lb8g" pod "3199ec19-a07d-4225-a9d8-1926f089ae6b" (UID: "3199ec19-a07d-4225-a9d8-1926f089ae6b"). InnerVolumeSpecName "kube-api-access-7lb8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:33:29.529755 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:29.529715    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3199ec19-a07d-4225-a9d8-1926f089ae6b-maas-api-xks-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:29.529755 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:29.529746    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lb8g\" (UniqueName: \"kubernetes.io/projected/3199ec19-a07d-4225-a9d8-1926f089ae6b-kube-api-access-7lb8g\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:30.180645 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:30.180600    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7b9df-2lphz" event={"ID":"3199ec19-a07d-4225-a9d8-1926f089ae6b","Type":"ContainerDied","Data":"809075724f6936f2254e24ac96769f800dcdc493b56cbc9cbb620f5abf07c88c"}
Oct 30 17:33:30.180850 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:30.180671    4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="809075724f6936f2254e24ac96769f800dcdc493b56cbc9cbb620f5abf07c88c"
Oct 30 17:33:30.180850 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:30.180749    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7b9df-2lphz"
Oct 30 17:33:31.132456 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.132090    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:33:31.201001 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.200966    4433 generic.go:356] "Generic (PLEG): container finished" podID="6906457b-d282-4603-8e72-edd173a4afa7" containerID="549a6c08fedba5e518e75ca00986b095623032ba88e99849abc50fc4d6ff2aaa" exitCode=0
Oct 30 17:33:31.201193 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.201054    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q" event={"ID":"6906457b-d282-4603-8e72-edd173a4afa7","Type":"ContainerDied","Data":"549a6c08fedba5e518e75ca00986b095623032ba88e99849abc50fc4d6ff2aaa"}
Oct 30 17:33:31.201193 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.201105    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q" event={"ID":"6906457b-d282-4603-8e72-edd173a4afa7","Type":"ContainerDied","Data":"dbb30f4d40fc09e4d9605d2e1f83ef9601dbbc3dfbe91b0d0e25fa7776def0a0"}
Oct 30 17:33:31.201193 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.201134    4433 scope.go:122] "RemoveContainer" containerID="549a6c08fedba5e518e75ca00986b095623032ba88e99849abc50fc4d6ff2aaa"
Oct 30 17:33:31.201377 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.201315    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"
Oct 30 17:33:31.225443 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.225383    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6906457b-d282-4603-8e72-edd173a4afa7-kube-api-access-rqzrf\" (UniqueName: \"kubernetes.io/projected/6906457b-d282-4603-8e72-edd173a4afa7-kube-api-access-rqzrf\") pod \"6906457b-d282-4603-8e72-edd173a4afa7\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") "
Oct 30 17:33:31.225629 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.225581    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6906457b-d282-4603-8e72-edd173a4afa7-istio-token\" (UniqueName: \"kubernetes.io/projected/6906457b-d282-4603-8e72-edd173a4afa7-istio-token\") pod \"6906457b-d282-4603-8e72-edd173a4afa7\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") "
Oct 30 17:33:31.225701 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.225665    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6906457b-d282-4603-8e72-edd173a4afa7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6906457b-d282-4603-8e72-edd173a4afa7-istio-data\") pod \"6906457b-d282-4603-8e72-edd173a4afa7\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") "
Oct 30 17:33:31.225767 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.225722    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6906457b-d282-4603-8e72-edd173a4afa7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6906457b-d282-4603-8e72-edd173a4afa7-istio-envoy\") pod \"6906457b-d282-4603-8e72-edd173a4afa7\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") "
Oct 30 17:33:31.225821 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.225766    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6906457b-d282-4603-8e72-edd173a4afa7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6906457b-d282-4603-8e72-edd173a4afa7-workload-certs\") pod \"6906457b-d282-4603-8e72-edd173a4afa7\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") "
Oct 30 17:33:31.225821 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.225806    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6906457b-d282-4603-8e72-edd173a4afa7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6906457b-d282-4603-8e72-edd173a4afa7-istio-podinfo\") pod \"6906457b-d282-4603-8e72-edd173a4afa7\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") "
Oct 30 17:33:31.225926 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.225883    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6906457b-d282-4603-8e72-edd173a4afa7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6906457b-d282-4603-8e72-edd173a4afa7-istiod-ca-cert\") pod \"6906457b-d282-4603-8e72-edd173a4afa7\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") "
Oct 30 17:33:31.225969 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.225950    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6906457b-d282-4603-8e72-edd173a4afa7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6906457b-d282-4603-8e72-edd173a4afa7-workload-socket\") pod \"6906457b-d282-4603-8e72-edd173a4afa7\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") "
Oct 30 17:33:31.226149 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.226127    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6906457b-d282-4603-8e72-edd173a4afa7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6906457b-d282-4603-8e72-edd173a4afa7-credential-socket\") pod \"6906457b-d282-4603-8e72-edd173a4afa7\" (UID: \"6906457b-d282-4603-8e72-edd173a4afa7\") "
Oct 30 17:33:31.227162 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.227132    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6906457b-d282-4603-8e72-edd173a4afa7-credential-socket" pod "6906457b-d282-4603-8e72-edd173a4afa7" (UID: "6906457b-d282-4603-8e72-edd173a4afa7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:33:31.227349 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.227331    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6906457b-d282-4603-8e72-edd173a4afa7-workload-certs" pod "6906457b-d282-4603-8e72-edd173a4afa7" (UID: "6906457b-d282-4603-8e72-edd173a4afa7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:33:31.228960 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.228748    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6906457b-d282-4603-8e72-edd173a4afa7-istiod-ca-cert" pod "6906457b-d282-4603-8e72-edd173a4afa7" (UID: "6906457b-d282-4603-8e72-edd173a4afa7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:33:31.229386 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.229110    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6906457b-d282-4603-8e72-edd173a4afa7-workload-socket" pod "6906457b-d282-4603-8e72-edd173a4afa7" (UID: "6906457b-d282-4603-8e72-edd173a4afa7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:33:31.230035 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.230010    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6906457b-d282-4603-8e72-edd173a4afa7-istio-podinfo" pod "6906457b-d282-4603-8e72-edd173a4afa7" (UID: "6906457b-d282-4603-8e72-edd173a4afa7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:33:31.230241 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.230222    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6906457b-d282-4603-8e72-edd173a4afa7-istio-data" pod "6906457b-d282-4603-8e72-edd173a4afa7" (UID: "6906457b-d282-4603-8e72-edd173a4afa7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:33:31.231656 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.231637    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6906457b-d282-4603-8e72-edd173a4afa7-istio-envoy" pod "6906457b-d282-4603-8e72-edd173a4afa7" (UID: "6906457b-d282-4603-8e72-edd173a4afa7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:33:31.234935 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.234898    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6906457b-d282-4603-8e72-edd173a4afa7-istio-token" pod "6906457b-d282-4603-8e72-edd173a4afa7" (UID: "6906457b-d282-4603-8e72-edd173a4afa7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:33:31.242541 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.242398    4433 scope.go:122] "RemoveContainer" containerID="478a9cff5355cc0e9b03e6c60c9427195fa3eb2851f17266a5c16756bd379c8a"
Oct 30 17:33:31.243188 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.242714    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6906457b-d282-4603-8e72-edd173a4afa7-kube-api-access-rqzrf" pod "6906457b-d282-4603-8e72-edd173a4afa7" (UID: "6906457b-d282-4603-8e72-edd173a4afa7"). InnerVolumeSpecName "kube-api-access-rqzrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:33:31.285671 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.279585    4433 scope.go:122] "RemoveContainer" containerID="549a6c08fedba5e518e75ca00986b095623032ba88e99849abc50fc4d6ff2aaa"
Oct 30 17:33:31.285671 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:33:31.279991    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"549a6c08fedba5e518e75ca00986b095623032ba88e99849abc50fc4d6ff2aaa\": container with ID starting with 549a6c08fedba5e518e75ca00986b095623032ba88e99849abc50fc4d6ff2aaa not found: ID does not exist" containerID="549a6c08fedba5e518e75ca00986b095623032ba88e99849abc50fc4d6ff2aaa"
Oct 30 17:33:31.285671 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.280022    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"549a6c08fedba5e518e75ca00986b095623032ba88e99849abc50fc4d6ff2aaa"} err="failed to get container status \"549a6c08fedba5e518e75ca00986b095623032ba88e99849abc50fc4d6ff2aaa\": rpc error: code = NotFound desc = could not find container \"549a6c08fedba5e518e75ca00986b095623032ba88e99849abc50fc4d6ff2aaa\": container with ID starting with 549a6c08fedba5e518e75ca00986b095623032ba88e99849abc50fc4d6ff2aaa not found: ID does not exist"
Oct 30 17:33:31.285671 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.280041    4433 scope.go:122] "RemoveContainer" containerID="478a9cff5355cc0e9b03e6c60c9427195fa3eb2851f17266a5c16756bd379c8a"
Oct 30 17:33:31.285671 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:33:31.283753    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"478a9cff5355cc0e9b03e6c60c9427195fa3eb2851f17266a5c16756bd379c8a\": container with ID starting with 478a9cff5355cc0e9b03e6c60c9427195fa3eb2851f17266a5c16756bd379c8a not found: ID does not exist" containerID="478a9cff5355cc0e9b03e6c60c9427195fa3eb2851f17266a5c16756bd379c8a"
Oct 30 17:33:31.285671 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.283792    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"478a9cff5355cc0e9b03e6c60c9427195fa3eb2851f17266a5c16756bd379c8a"} err="failed to get container status \"478a9cff5355cc0e9b03e6c60c9427195fa3eb2851f17266a5c16756bd379c8a\": rpc error: code = NotFound desc = could not find container \"478a9cff5355cc0e9b03e6c60c9427195fa3eb2851f17266a5c16756bd379c8a\": container with ID starting with 478a9cff5355cc0e9b03e6c60c9427195fa3eb2851f17266a5c16756bd379c8a not found: ID does not exist"
Oct 30 17:33:31.328795 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.328634    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6906457b-d282-4603-8e72-edd173a4afa7-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:31.328795 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.328666    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6906457b-d282-4603-8e72-edd173a4afa7-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:31.328795 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.328679    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6906457b-d282-4603-8e72-edd173a4afa7-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:31.328795 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.328693    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6906457b-d282-4603-8e72-edd173a4afa7-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:31.328795 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.328707    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6906457b-d282-4603-8e72-edd173a4afa7-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:31.328795 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.328721    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6906457b-d282-4603-8e72-edd173a4afa7-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:31.328795 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.328736    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6906457b-d282-4603-8e72-edd173a4afa7-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:31.328795 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.328751    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6906457b-d282-4603-8e72-edd173a4afa7-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:31.328795 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.328766    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqzrf\" (UniqueName: \"kubernetes.io/projected/6906457b-d282-4603-8e72-edd173a4afa7-kube-api-access-rqzrf\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:31.579589 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.579484    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"]
Oct 30 17:33:31.583679 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.583650    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q"]
Oct 30 17:33:31.697031 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:31.696995    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6906457b-d282-4603-8e72-edd173a4afa7" path="/var/lib/kubelet/pods/6906457b-d282-4603-8e72-edd173a4afa7/volumes"
Oct 30 17:33:32.115377 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:32.115337    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:33:32.115606 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:32.115396    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-001a96-openshift-default-7776656fbc-sxr9q" podUID="6906457b-d282-4603-8e72-edd173a4afa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:33.718797 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:33.718763    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e334447d-xlmzn"]
Oct 30 17:33:33.719453 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:33.719437    4433 cpu_manager.go:414] "containerMap: removing container" podUID="6906457b-d282-4603-8e72-edd173a4afa7" containerName="istio-proxy"
Oct 30 17:33:33.719509 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:33.719454    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6906457b-d282-4603-8e72-edd173a4afa7" containerName="istio-proxy"
Oct 30 17:33:33.719509 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:33.719469    4433 cpu_manager.go:414] "containerMap: removing container" podUID="6906457b-d282-4603-8e72-edd173a4afa7" containerName="istio-proxy"
Oct 30 17:33:33.719509 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:33.719475    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6906457b-d282-4603-8e72-edd173a4afa7" containerName="istio-proxy"
Oct 30 17:33:33.719509 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:33.719494    4433 cpu_manager.go:414] "containerMap: removing container" podUID="3199ec19-a07d-4225-a9d8-1926f089ae6b" containerName="revoke-keys"
Oct 30 17:33:33.719509 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:33.719499    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3199ec19-a07d-4225-a9d8-1926f089ae6b" containerName="revoke-keys"
Oct 30 17:33:33.719671 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:33.719594    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="6906457b-d282-4603-8e72-edd173a4afa7" containerName="istio-proxy"
Oct 30 17:33:33.719671 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:33.719602    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="6906457b-d282-4603-8e72-edd173a4afa7" containerName="istio-proxy"
Oct 30 17:33:33.719671 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:33.719611    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="3199ec19-a07d-4225-a9d8-1926f089ae6b" containerName="revoke-keys"
Oct 30 17:33:33.723241 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:33.723220    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e334447d-xlmzn"
Oct 30 17:33:33.768429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:33.767463    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\""
Oct 30 17:33:33.768429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:33.767638    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:33:33.785436 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:33.785379    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e334447d-xlmzn"]
Oct 30 17:33:33.807809 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:33.807777    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec44098-d2da-43d5-a3da-0667808a27cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e334447d-xlmzn\" (UID: \"fec44098-d2da-43d5-a3da-0667808a27cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e334447d-xlmzn"
Oct 30 17:33:33.807993 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:33.807845    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec44098-d2da-43d5-a3da-0667808a27cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e334447d-xlmzn\" (UID: \"fec44098-d2da-43d5-a3da-0667808a27cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e334447d-xlmzn"
Oct 30 17:33:33.808326 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:33.808298    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swh49\" (UniqueName: \"kubernetes.io/projected/fec44098-d2da-43d5-a3da-0667808a27cf-kube-api-access-swh49\") pod \"maas-api-revoke-sub-e334447d-xlmzn\" (UID: \"fec44098-d2da-43d5-a3da-0667808a27cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e334447d-xlmzn"
Oct 30 17:33:33.925096 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:33.923983    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swh49\" (UniqueName: \"kubernetes.io/projected/fec44098-d2da-43d5-a3da-0667808a27cf-kube-api-access-swh49\") pod \"maas-api-revoke-sub-e334447d-xlmzn\" (UID: \"fec44098-d2da-43d5-a3da-0667808a27cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e334447d-xlmzn"
Oct 30 17:33:33.938426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:33.937534    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec44098-d2da-43d5-a3da-0667808a27cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e334447d-xlmzn\" (UID: \"fec44098-d2da-43d5-a3da-0667808a27cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e334447d-xlmzn"
Oct 30 17:33:33.938426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:33.936458    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec44098-d2da-43d5-a3da-0667808a27cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e334447d-xlmzn\" (UID: \"fec44098-d2da-43d5-a3da-0667808a27cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e334447d-xlmzn"
Oct 30 17:33:33.945929 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:33.944640    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec44098-d2da-43d5-a3da-0667808a27cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e334447d-xlmzn\" (UID: \"fec44098-d2da-43d5-a3da-0667808a27cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e334447d-xlmzn"
Oct 30 17:33:33.969477 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:33.962449    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec44098-d2da-43d5-a3da-0667808a27cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e334447d-xlmzn\" (UID: \"fec44098-d2da-43d5-a3da-0667808a27cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e334447d-xlmzn"
Oct 30 17:33:33.979479 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:33.977227    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swh49\" (UniqueName: \"kubernetes.io/projected/fec44098-d2da-43d5-a3da-0667808a27cf-kube-api-access-swh49\") pod \"maas-api-revoke-sub-e334447d-xlmzn\" (UID: \"fec44098-d2da-43d5-a3da-0667808a27cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e334447d-xlmzn"
Oct 30 17:33:34.038039 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:34.038001    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e334447d-xlmzn"
Oct 30 17:33:34.384059 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:34.384032    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e334447d-xlmzn"]
Oct 30 17:33:34.386612 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:33:34.386584    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec44098_d2da_43d5_a3da_0667808a27cf.slice/crio-954229ca2e76bfe502facc5b818344924c7d684dea0175fea6ad07bbbabbb9b3 WatchSource:0}: Error finding container 954229ca2e76bfe502facc5b818344924c7d684dea0175fea6ad07bbbabbb9b3: Status 404 returned error can't find the container with id 954229ca2e76bfe502facc5b818344924c7d684dea0175fea6ad07bbbabbb9b3
Oct 30 17:33:35.233425 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:35.233376    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e334447d-xlmzn" event={"ID":"fec44098-d2da-43d5-a3da-0667808a27cf","Type":"ContainerStarted","Data":"45856953bdd048ca79b2510fd0fbbd7381e72ecbbd5397d1a9d88e70d1ac158b"}
Oct 30 17:33:35.233830 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:35.233456    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e334447d-xlmzn" event={"ID":"fec44098-d2da-43d5-a3da-0667808a27cf","Type":"ContainerStarted","Data":"954229ca2e76bfe502facc5b818344924c7d684dea0175fea6ad07bbbabbb9b3"}
Oct 30 17:33:35.280558 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:35.280495    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e334447d-xlmzn" podStartSLOduration=2.28047867 podStartE2EDuration="2.28047867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:35.27807048 +0000 UTC m=+2791.712972690" watchObservedRunningTime="2026-10-30 17:33:35.28047867 +0000 UTC m=+2791.715380877"
Oct 30 17:33:36.269430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:36.259709    4433 generic.go:356] "Generic (PLEG): container finished" podID="fec44098-d2da-43d5-a3da-0667808a27cf" containerID="45856953bdd048ca79b2510fd0fbbd7381e72ecbbd5397d1a9d88e70d1ac158b" exitCode=0
Oct 30 17:33:36.269430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:36.260143    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e334447d-xlmzn" event={"ID":"fec44098-d2da-43d5-a3da-0667808a27cf","Type":"ContainerDied","Data":"45856953bdd048ca79b2510fd0fbbd7381e72ecbbd5397d1a9d88e70d1ac158b"}
Oct 30 17:33:37.591761 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:37.591708    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e334447d-xlmzn"
Oct 30 17:33:37.661642 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:37.661605    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fec44098-d2da-43d5-a3da-0667808a27cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec44098-d2da-43d5-a3da-0667808a27cf-maas-api-xks-ca\") pod \"fec44098-d2da-43d5-a3da-0667808a27cf\" (UID: \"fec44098-d2da-43d5-a3da-0667808a27cf\") "
Oct 30 17:33:37.661832 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:37.661675    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fec44098-d2da-43d5-a3da-0667808a27cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec44098-d2da-43d5-a3da-0667808a27cf-maas-api-openshift-ca\") pod \"fec44098-d2da-43d5-a3da-0667808a27cf\" (UID: \"fec44098-d2da-43d5-a3da-0667808a27cf\") "
Oct 30 17:33:37.661832 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:37.661706    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fec44098-d2da-43d5-a3da-0667808a27cf-kube-api-access-swh49\" (UniqueName: \"kubernetes.io/projected/fec44098-d2da-43d5-a3da-0667808a27cf-kube-api-access-swh49\") pod \"fec44098-d2da-43d5-a3da-0667808a27cf\" (UID: \"fec44098-d2da-43d5-a3da-0667808a27cf\") "
Oct 30 17:33:37.662150 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:37.662119    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fec44098-d2da-43d5-a3da-0667808a27cf-maas-api-openshift-ca" pod "fec44098-d2da-43d5-a3da-0667808a27cf" (UID: "fec44098-d2da-43d5-a3da-0667808a27cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:33:37.662606 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:37.662581    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec44098-d2da-43d5-a3da-0667808a27cf-maas-api-openshift-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:37.664158 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:37.664134    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec44098-d2da-43d5-a3da-0667808a27cf-kube-api-access-swh49" pod "fec44098-d2da-43d5-a3da-0667808a27cf" (UID: "fec44098-d2da-43d5-a3da-0667808a27cf"). InnerVolumeSpecName "kube-api-access-swh49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:33:37.664437 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:37.664388    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fec44098-d2da-43d5-a3da-0667808a27cf-maas-api-xks-ca" pod "fec44098-d2da-43d5-a3da-0667808a27cf" (UID: "fec44098-d2da-43d5-a3da-0667808a27cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:33:37.766424 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:37.764805    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec44098-d2da-43d5-a3da-0667808a27cf-maas-api-xks-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:37.766424 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:37.764941    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swh49\" (UniqueName: \"kubernetes.io/projected/fec44098-d2da-43d5-a3da-0667808a27cf-kube-api-access-swh49\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:38.287325 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:38.287295    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e334447d-xlmzn"
Oct 30 17:33:38.287558 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:38.287332    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e334447d-xlmzn" event={"ID":"fec44098-d2da-43d5-a3da-0667808a27cf","Type":"ContainerDied","Data":"954229ca2e76bfe502facc5b818344924c7d684dea0175fea6ad07bbbabbb9b3"}
Oct 30 17:33:38.287558 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:38.287387    4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="954229ca2e76bfe502facc5b818344924c7d684dea0175fea6ad07bbbabbb9b3"
Oct 30 17:33:39.213604 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:39.213563    4433 patch_prober.go:29] interesting pod/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:33:39.214104 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:39.213640    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m" podUID="8b410f33-6553-4f8d-972f-b21558c4db1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:33:39.680621 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:39.680551    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41cb25-rbqcp"]
Oct 30 17:33:39.685344 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:39.685312    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41cb25-rbqcp"]
Oct 30 17:33:39.698064 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:39.698034    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="506303c1-2a19-4487-a657-c10901f0c16f" path="/var/lib/kubelet/pods/506303c1-2a19-4487-a657-c10901f0c16f/volumes"
Oct 30 17:33:42.236120 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:42.236083    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2d985-bw87p"]
Oct 30 17:33:42.246915 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:42.246886    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2d985-bw87p"]
Oct 30 17:33:42.461167 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:42.461128    4433 generic.go:356] "Generic (PLEG): container finished" podID="8b410f33-6553-4f8d-972f-b21558c4db1a" containerID="cc9c266c294d99efd5fd767e042e7c54ef42b46fe7abd8d39d868139a859c176" exitCode=0
Oct 30 17:33:42.461348 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:42.461198    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m" event={"ID":"8b410f33-6553-4f8d-972f-b21558c4db1a","Type":"ContainerDied","Data":"cc9c266c294d99efd5fd767e042e7c54ef42b46fe7abd8d39d868139a859c176"}
Oct 30 17:33:42.461681 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:42.461656    4433 scope.go:122] "RemoveContainer" containerID="cc9c266c294d99efd5fd767e042e7c54ef42b46fe7abd8d39d868139a859c176"
Oct 30 17:33:42.463427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:42.463381    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:33:42.463536 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:42.463482    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:33:42.463611 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:42.463532    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:33:43.484131 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:43.484096    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m" event={"ID":"8b410f33-6553-4f8d-972f-b21558c4db1a","Type":"ContainerStarted","Data":"9486e56ff5604bb9dc985aedae9cceb6fe908780dac53f09b8b5df8b1d2cc350"}
Oct 30 17:33:43.696155 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:43.696124    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fa2b418-d47e-4f0c-b0f5-36aa9dd14030" path="/var/lib/kubelet/pods/5fa2b418-d47e-4f0c-b0f5-36aa9dd14030/volumes"
Oct 30 17:33:44.325515 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:44.325482    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:33:44.325784 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:44.325760    4433 patch_prober.go:29] interesting pod/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 17:33:44.325848 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:44.325812    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m" podUID="8b410f33-6553-4f8d-972f-b21558c4db1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 17:33:45.325446 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:45.325388    4433 patch_prober.go:29] interesting pod/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 17:33:45.325860 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:45.325466    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m" podUID="8b410f33-6553-4f8d-972f-b21558c4db1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 17:33:46.325192 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:46.325148    4433 patch_prober.go:29] interesting pod/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 17:33:46.325362 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:46.325207    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m" podUID="8b410f33-6553-4f8d-972f-b21558c4db1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 17:33:47.005206 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:47.003707    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"]
Oct 30 17:33:47.005206 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:47.003977    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p" podUID="988a3f4c-5f27-474e-a55a-38f5c555d374" containerName="istio-proxy" containerID="cri-o://7d0a57f4cd1b6a8c2b987ba8ac99ffa196721f72a65ce1fdc9b0237f5e5286f8" gracePeriod=30
Oct 30 17:33:47.342492 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:47.336802    4433 patch_prober.go:29] interesting pod/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 17:33:47.342492 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:47.336863    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m" podUID="8b410f33-6553-4f8d-972f-b21558c4db1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 17:33:48.326927 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:48.326890    4433 patch_prober.go:29] interesting pod/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 17:33:48.327326 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:48.326946    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m" podUID="8b410f33-6553-4f8d-972f-b21558c4db1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 17:33:48.626851 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:48.626758    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"]
Oct 30 17:33:48.627031 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:48.626968    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh" podUID="1edce449-0010-4d41-908e-db02f1c508f9" containerName="istio-proxy" containerID="cri-o://801a4a946d4fd753424aaaee540ca2cacb5841d9e397e04d5d99de630eef6e5a" gracePeriod=30
Oct 30 17:33:49.396912 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:49.396867    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:33:49.398214 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:49.398189    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:33:49.398807 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:49.398787    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:33:52.325700 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.325665    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"
Oct 30 17:33:52.405483 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.405446    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/988a3f4c-5f27-474e-a55a-38f5c555d374-kube-api-access-rr9g5\" (UniqueName: \"kubernetes.io/projected/988a3f4c-5f27-474e-a55a-38f5c555d374-kube-api-access-rr9g5\") pod \"988a3f4c-5f27-474e-a55a-38f5c555d374\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") "
Oct 30 17:33:52.405681 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.405551    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/988a3f4c-5f27-474e-a55a-38f5c555d374-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/988a3f4c-5f27-474e-a55a-38f5c555d374-istio-envoy\") pod \"988a3f4c-5f27-474e-a55a-38f5c555d374\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") "
Oct 30 17:33:52.405681 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.405637    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/988a3f4c-5f27-474e-a55a-38f5c555d374-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/988a3f4c-5f27-474e-a55a-38f5c555d374-workload-certs\") pod \"988a3f4c-5f27-474e-a55a-38f5c555d374\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") "
Oct 30 17:33:52.405809 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.405700    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/988a3f4c-5f27-474e-a55a-38f5c555d374-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/988a3f4c-5f27-474e-a55a-38f5c555d374-credential-socket\") pod \"988a3f4c-5f27-474e-a55a-38f5c555d374\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") "
Oct 30 17:33:52.405809 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.405771    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/988a3f4c-5f27-474e-a55a-38f5c555d374-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/988a3f4c-5f27-474e-a55a-38f5c555d374-istio-podinfo\") pod \"988a3f4c-5f27-474e-a55a-38f5c555d374\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") "
Oct 30 17:33:52.405928 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.405822    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/988a3f4c-5f27-474e-a55a-38f5c555d374-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/988a3f4c-5f27-474e-a55a-38f5c555d374-workload-socket\") pod \"988a3f4c-5f27-474e-a55a-38f5c555d374\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") "
Oct 30 17:33:52.405986 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.405943    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/988a3f4c-5f27-474e-a55a-38f5c555d374-istio-token\" (UniqueName: \"kubernetes.io/projected/988a3f4c-5f27-474e-a55a-38f5c555d374-istio-token\") pod \"988a3f4c-5f27-474e-a55a-38f5c555d374\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") "
Oct 30 17:33:52.406044 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.405993    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/988a3f4c-5f27-474e-a55a-38f5c555d374-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/988a3f4c-5f27-474e-a55a-38f5c555d374-istio-data\") pod \"988a3f4c-5f27-474e-a55a-38f5c555d374\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") "
Oct 30 17:33:52.406044 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.405990    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/988a3f4c-5f27-474e-a55a-38f5c555d374-workload-certs" pod "988a3f4c-5f27-474e-a55a-38f5c555d374" (UID: "988a3f4c-5f27-474e-a55a-38f5c555d374"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:33:52.406044 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.405996    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/988a3f4c-5f27-474e-a55a-38f5c555d374-credential-socket" pod "988a3f4c-5f27-474e-a55a-38f5c555d374" (UID: "988a3f4c-5f27-474e-a55a-38f5c555d374"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:33:52.406044 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.406033    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/988a3f4c-5f27-474e-a55a-38f5c555d374-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/988a3f4c-5f27-474e-a55a-38f5c555d374-istiod-ca-cert\") pod \"988a3f4c-5f27-474e-a55a-38f5c555d374\" (UID: \"988a3f4c-5f27-474e-a55a-38f5c555d374\") "
Oct 30 17:33:52.406425 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.406235    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/988a3f4c-5f27-474e-a55a-38f5c555d374-istio-data" pod "988a3f4c-5f27-474e-a55a-38f5c555d374" (UID: "988a3f4c-5f27-474e-a55a-38f5c555d374"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:33:52.406526 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.406419    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/988a3f4c-5f27-474e-a55a-38f5c555d374-workload-socket" pod "988a3f4c-5f27-474e-a55a-38f5c555d374" (UID: "988a3f4c-5f27-474e-a55a-38f5c555d374"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:33:52.406798 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.406773    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/988a3f4c-5f27-474e-a55a-38f5c555d374-istiod-ca-cert" pod "988a3f4c-5f27-474e-a55a-38f5c555d374" (UID: "988a3f4c-5f27-474e-a55a-38f5c555d374"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:33:52.406956 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.406868    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/988a3f4c-5f27-474e-a55a-38f5c555d374-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:52.407076 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.406968    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/988a3f4c-5f27-474e-a55a-38f5c555d374-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:52.407076 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.406984    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/988a3f4c-5f27-474e-a55a-38f5c555d374-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:52.407076 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.406999    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/988a3f4c-5f27-474e-a55a-38f5c555d374-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:52.408080 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.408055    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/988a3f4c-5f27-474e-a55a-38f5c555d374-kube-api-access-rr9g5" pod "988a3f4c-5f27-474e-a55a-38f5c555d374" (UID: "988a3f4c-5f27-474e-a55a-38f5c555d374"). InnerVolumeSpecName "kube-api-access-rr9g5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:33:52.408352 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.408330    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/988a3f4c-5f27-474e-a55a-38f5c555d374-istio-envoy" pod "988a3f4c-5f27-474e-a55a-38f5c555d374" (UID: "988a3f4c-5f27-474e-a55a-38f5c555d374"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:33:52.408698 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.408673    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/988a3f4c-5f27-474e-a55a-38f5c555d374-istio-token" pod "988a3f4c-5f27-474e-a55a-38f5c555d374" (UID: "988a3f4c-5f27-474e-a55a-38f5c555d374"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:33:52.408851 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.408831    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/988a3f4c-5f27-474e-a55a-38f5c555d374-istio-podinfo" pod "988a3f4c-5f27-474e-a55a-38f5c555d374" (UID: "988a3f4c-5f27-474e-a55a-38f5c555d374"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:33:52.508659 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.508624    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/988a3f4c-5f27-474e-a55a-38f5c555d374-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:52.508659 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.508652    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/988a3f4c-5f27-474e-a55a-38f5c555d374-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:52.508659 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.508661    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/988a3f4c-5f27-474e-a55a-38f5c555d374-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:52.508659 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.508670    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr9g5\" (UniqueName: \"kubernetes.io/projected/988a3f4c-5f27-474e-a55a-38f5c555d374-kube-api-access-rr9g5\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:52.508932 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.508679    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/988a3f4c-5f27-474e-a55a-38f5c555d374-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:52.637465 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.637359    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3260f8-pws4g"]
Oct 30 17:33:52.638002 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.637987    4433 cpu_manager.go:414] "containerMap: removing container" podUID="988a3f4c-5f27-474e-a55a-38f5c555d374" containerName="istio-proxy"
Oct 30 17:33:52.638103 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.638004    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="988a3f4c-5f27-474e-a55a-38f5c555d374" containerName="istio-proxy"
Oct 30 17:33:52.638103 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.638037    4433 cpu_manager.go:414] "containerMap: removing container" podUID="fec44098-d2da-43d5-a3da-0667808a27cf" containerName="revoke-keys"
Oct 30 17:33:52.638103 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.638042    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fec44098-d2da-43d5-a3da-0667808a27cf" containerName="revoke-keys"
Oct 30 17:33:52.638199 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.638139    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="fec44098-d2da-43d5-a3da-0667808a27cf" containerName="revoke-keys"
Oct 30 17:33:52.638199 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.638152    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="988a3f4c-5f27-474e-a55a-38f5c555d374" containerName="istio-proxy"
Oct 30 17:33:52.642314 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.642286    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3260f8-pws4g"
Oct 30 17:33:52.661746 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.661720    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:33:52.661883 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.661738    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\""
Oct 30 17:33:52.665869 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.665845    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3260f8-pws4g"]
Oct 30 17:33:52.711329 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.711285    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw8jw\" (UniqueName: \"kubernetes.io/projected/3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0-kube-api-access-pw8jw\") pod \"maas-api-revoke-keys-e2e-ipp-b-3260f8-pws4g\" (UID: \"3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3260f8-pws4g"
Oct 30 17:33:52.711651 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.711624    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3260f8-pws4g\" (UID: \"3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3260f8-pws4g"
Oct 30 17:33:52.711827 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.711798    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3260f8-pws4g\" (UID: \"3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3260f8-pws4g"
Oct 30 17:33:52.813142 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.813104    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3260f8-pws4g\" (UID: \"3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3260f8-pws4g"
Oct 30 17:33:52.813310 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.813271    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw8jw\" (UniqueName: \"kubernetes.io/projected/3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0-kube-api-access-pw8jw\") pod \"maas-api-revoke-keys-e2e-ipp-b-3260f8-pws4g\" (UID: \"3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3260f8-pws4g"
Oct 30 17:33:52.813479 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.813457    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3260f8-pws4g\" (UID: \"3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3260f8-pws4g"
Oct 30 17:33:52.813891 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.813874    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3260f8-pws4g\" (UID: \"3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3260f8-pws4g"
Oct 30 17:33:52.815907 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.815884    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3260f8-pws4g\" (UID: \"3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3260f8-pws4g"
Oct 30 17:33:52.835686 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.835664    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw8jw\" (UniqueName: \"kubernetes.io/projected/3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0-kube-api-access-pw8jw\") pod \"maas-api-revoke-keys-e2e-ipp-b-3260f8-pws4g\" (UID: \"3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3260f8-pws4g"
Oct 30 17:33:52.864310 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.864280    4433 generic.go:356] "Generic (PLEG): container finished" podID="988a3f4c-5f27-474e-a55a-38f5c555d374" containerID="7d0a57f4cd1b6a8c2b987ba8ac99ffa196721f72a65ce1fdc9b0237f5e5286f8" exitCode=0
Oct 30 17:33:52.864475 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.864328    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"
Oct 30 17:33:52.864475 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.864371    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p" event={"ID":"988a3f4c-5f27-474e-a55a-38f5c555d374","Type":"ContainerDied","Data":"7d0a57f4cd1b6a8c2b987ba8ac99ffa196721f72a65ce1fdc9b0237f5e5286f8"}
Oct 30 17:33:52.864475 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.864449    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p" event={"ID":"988a3f4c-5f27-474e-a55a-38f5c555d374","Type":"ContainerDied","Data":"249cafe4120350287f8f7e2098d7c02d31445a7d89828b3832f1bf0fe33888c7"}
Oct 30 17:33:52.864680 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.864479    4433 scope.go:122] "RemoveContainer" containerID="7d0a57f4cd1b6a8c2b987ba8ac99ffa196721f72a65ce1fdc9b0237f5e5286f8"
Oct 30 17:33:52.875372 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.875355    4433 scope.go:122] "RemoveContainer" containerID="7d0a57f4cd1b6a8c2b987ba8ac99ffa196721f72a65ce1fdc9b0237f5e5286f8"
Oct 30 17:33:52.875659 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:33:52.875641    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d0a57f4cd1b6a8c2b987ba8ac99ffa196721f72a65ce1fdc9b0237f5e5286f8\": container with ID starting with 7d0a57f4cd1b6a8c2b987ba8ac99ffa196721f72a65ce1fdc9b0237f5e5286f8 not found: ID does not exist" containerID="7d0a57f4cd1b6a8c2b987ba8ac99ffa196721f72a65ce1fdc9b0237f5e5286f8"
Oct 30 17:33:52.875719 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.875670    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d0a57f4cd1b6a8c2b987ba8ac99ffa196721f72a65ce1fdc9b0237f5e5286f8"} err="failed to get container status \"7d0a57f4cd1b6a8c2b987ba8ac99ffa196721f72a65ce1fdc9b0237f5e5286f8\": rpc error: code = NotFound desc = could not find container \"7d0a57f4cd1b6a8c2b987ba8ac99ffa196721f72a65ce1fdc9b0237f5e5286f8\": container with ID starting with 7d0a57f4cd1b6a8c2b987ba8ac99ffa196721f72a65ce1fdc9b0237f5e5286f8 not found: ID does not exist"
Oct 30 17:33:52.916328 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.916258    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"]
Oct 30 17:33:52.919721 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.919697    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-bz66p"]
Oct 30 17:33:52.956463 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:52.956433    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3260f8-pws4g"
Oct 30 17:33:53.269629 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:53.269595    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3260f8-pws4g"]
Oct 30 17:33:53.271077 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:33:53.271050    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c1cc3f6_c25c_4c55_b8f8_76fa94b25aa0.slice/crio-b023365a98a3f5298f51cce300589c51d4ab0f0d1c18b6ee10524042a5537e58 WatchSource:0}: Error finding container b023365a98a3f5298f51cce300589c51d4ab0f0d1c18b6ee10524042a5537e58: Status 404 returned error can't find the container with id b023365a98a3f5298f51cce300589c51d4ab0f0d1c18b6ee10524042a5537e58
Oct 30 17:33:53.696181 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:53.696150    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="988a3f4c-5f27-474e-a55a-38f5c555d374" path="/var/lib/kubelet/pods/988a3f4c-5f27-474e-a55a-38f5c555d374/volumes"
Oct 30 17:33:53.873972 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:53.873890    4433 generic.go:356] "Generic (PLEG): container finished" podID="1edce449-0010-4d41-908e-db02f1c508f9" containerID="801a4a946d4fd753424aaaee540ca2cacb5841d9e397e04d5d99de630eef6e5a" exitCode=0
Oct 30 17:33:53.873972 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:53.873959    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh" event={"ID":"1edce449-0010-4d41-908e-db02f1c508f9","Type":"ContainerDied","Data":"801a4a946d4fd753424aaaee540ca2cacb5841d9e397e04d5d99de630eef6e5a"}
Oct 30 17:33:53.875604 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:53.875573    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3260f8-pws4g" event={"ID":"3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0","Type":"ContainerStarted","Data":"ff52d06aece677ba41bd21974a28579ca10e6dbe4056967abb30e8de21b60f3e"}
Oct 30 17:33:53.875753 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:53.875617    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3260f8-pws4g" event={"ID":"3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0","Type":"ContainerStarted","Data":"b023365a98a3f5298f51cce300589c51d4ab0f0d1c18b6ee10524042a5537e58"}
Oct 30 17:33:53.908337 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:53.908311    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"
Oct 30 17:33:53.917891 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:53.917845    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3260f8-pws4g" podStartSLOduration=1.917832721 podStartE2EDuration="1.917832721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:53.914279114 +0000 UTC m=+2810.349181319" watchObservedRunningTime="2026-10-30 17:33:53.917832721 +0000 UTC m=+2810.352734926"
Oct 30 17:33:53.929520 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:53.929494    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1edce449-0010-4d41-908e-db02f1c508f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1edce449-0010-4d41-908e-db02f1c508f9-istio-podinfo\") pod \"1edce449-0010-4d41-908e-db02f1c508f9\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") "
Oct 30 17:33:53.929696 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:53.929591    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1edce449-0010-4d41-908e-db02f1c508f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1edce449-0010-4d41-908e-db02f1c508f9-istio-data\") pod \"1edce449-0010-4d41-908e-db02f1c508f9\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") "
Oct 30 17:33:53.929696 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:53.929641    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1edce449-0010-4d41-908e-db02f1c508f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1edce449-0010-4d41-908e-db02f1c508f9-workload-socket\") pod \"1edce449-0010-4d41-908e-db02f1c508f9\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") "
Oct 30 17:33:53.929787 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:53.929696    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1edce449-0010-4d41-908e-db02f1c508f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1edce449-0010-4d41-908e-db02f1c508f9-workload-certs\") pod \"1edce449-0010-4d41-908e-db02f1c508f9\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") "
Oct 30 17:33:53.929787 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:53.929774    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1edce449-0010-4d41-908e-db02f1c508f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1edce449-0010-4d41-908e-db02f1c508f9-istio-envoy\") pod \"1edce449-0010-4d41-908e-db02f1c508f9\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") "
Oct 30 17:33:53.929895 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:53.929807    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1edce449-0010-4d41-908e-db02f1c508f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1edce449-0010-4d41-908e-db02f1c508f9-credential-socket\") pod \"1edce449-0010-4d41-908e-db02f1c508f9\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") "
Oct 30 17:33:53.929958 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:53.929926    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1edce449-0010-4d41-908e-db02f1c508f9-istio-data" pod "1edce449-0010-4d41-908e-db02f1c508f9" (UID: "1edce449-0010-4d41-908e-db02f1c508f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:33:53.929958 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:53.929947    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1edce449-0010-4d41-908e-db02f1c508f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1edce449-0010-4d41-908e-db02f1c508f9-istiod-ca-cert\") pod \"1edce449-0010-4d41-908e-db02f1c508f9\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") "
Oct 30 17:33:53.930067 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:53.930003    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1edce449-0010-4d41-908e-db02f1c508f9-kube-api-access-ntx88\" (UniqueName: \"kubernetes.io/projected/1edce449-0010-4d41-908e-db02f1c508f9-kube-api-access-ntx88\") pod \"1edce449-0010-4d41-908e-db02f1c508f9\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") "
Oct 30 17:33:53.930067 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:53.929931    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1edce449-0010-4d41-908e-db02f1c508f9-workload-socket" pod "1edce449-0010-4d41-908e-db02f1c508f9" (UID: "1edce449-0010-4d41-908e-db02f1c508f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:33:53.930067 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:53.930028    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1edce449-0010-4d41-908e-db02f1c508f9-istio-token\" (UniqueName: \"kubernetes.io/projected/1edce449-0010-4d41-908e-db02f1c508f9-istio-token\") pod \"1edce449-0010-4d41-908e-db02f1c508f9\" (UID: \"1edce449-0010-4d41-908e-db02f1c508f9\") "
Oct 30 17:33:53.930230 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:53.930124    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1edce449-0010-4d41-908e-db02f1c508f9-credential-socket" pod "1edce449-0010-4d41-908e-db02f1c508f9" (UID: "1edce449-0010-4d41-908e-db02f1c508f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:33:53.930280 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:53.930263    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1edce449-0010-4d41-908e-db02f1c508f9-workload-certs" pod "1edce449-0010-4d41-908e-db02f1c508f9" (UID: "1edce449-0010-4d41-908e-db02f1c508f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:33:53.930386 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:53.930363    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1edce449-0010-4d41-908e-db02f1c508f9-istiod-ca-cert" pod "1edce449-0010-4d41-908e-db02f1c508f9" (UID: "1edce449-0010-4d41-908e-db02f1c508f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:33:53.931036 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:53.931012    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1edce449-0010-4d41-908e-db02f1c508f9-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:53.931184 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:53.931044    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1edce449-0010-4d41-908e-db02f1c508f9-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:53.931184 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:53.931060    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1edce449-0010-4d41-908e-db02f1c508f9-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:53.931184 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:53.931074    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1edce449-0010-4d41-908e-db02f1c508f9-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:53.931184 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:53.931089    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1edce449-0010-4d41-908e-db02f1c508f9-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:53.932052 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:53.932014    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1edce449-0010-4d41-908e-db02f1c508f9-istio-podinfo" pod "1edce449-0010-4d41-908e-db02f1c508f9" (UID: "1edce449-0010-4d41-908e-db02f1c508f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:33:53.932141 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:53.932084    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1edce449-0010-4d41-908e-db02f1c508f9-istio-envoy" pod "1edce449-0010-4d41-908e-db02f1c508f9" (UID: "1edce449-0010-4d41-908e-db02f1c508f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:33:53.932188 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:53.932166    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1edce449-0010-4d41-908e-db02f1c508f9-istio-token" pod "1edce449-0010-4d41-908e-db02f1c508f9" (UID: "1edce449-0010-4d41-908e-db02f1c508f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:33:53.932647 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:53.932629    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1edce449-0010-4d41-908e-db02f1c508f9-kube-api-access-ntx88" pod "1edce449-0010-4d41-908e-db02f1c508f9" (UID: "1edce449-0010-4d41-908e-db02f1c508f9"). InnerVolumeSpecName "kube-api-access-ntx88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:33:54.032736 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:54.032700    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntx88\" (UniqueName: \"kubernetes.io/projected/1edce449-0010-4d41-908e-db02f1c508f9-kube-api-access-ntx88\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:54.032736 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:54.032736    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1edce449-0010-4d41-908e-db02f1c508f9-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:54.032940 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:54.032752    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1edce449-0010-4d41-908e-db02f1c508f9-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:54.032940 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:54.032765    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1edce449-0010-4d41-908e-db02f1c508f9-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:54.884252 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:54.884206    4433 generic.go:356] "Generic (PLEG): container finished" podID="3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0" containerID="ff52d06aece677ba41bd21974a28579ca10e6dbe4056967abb30e8de21b60f3e" exitCode=0
Oct 30 17:33:54.884880 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:54.884852    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3260f8-pws4g" event={"ID":"3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0","Type":"ContainerDied","Data":"ff52d06aece677ba41bd21974a28579ca10e6dbe4056967abb30e8de21b60f3e"}
Oct 30 17:33:54.886612 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:54.886572    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh" event={"ID":"1edce449-0010-4d41-908e-db02f1c508f9","Type":"ContainerDied","Data":"ccf58336975ec8cdeb730406fbf0470229aab6f570810c05e84d912c30d25176"}
Oct 30 17:33:54.886755 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:54.886638    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"
Oct 30 17:33:54.886755 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:54.886663    4433 scope.go:122] "RemoveContainer" containerID="801a4a946d4fd753424aaaee540ca2cacb5841d9e397e04d5d99de630eef6e5a"
Oct 30 17:33:54.961420 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:54.961377    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"]
Oct 30 17:33:54.973154 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:54.973123    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-kjbqh"]
Oct 30 17:33:55.696905 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:55.696871    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1edce449-0010-4d41-908e-db02f1c508f9" path="/var/lib/kubelet/pods/1edce449-0010-4d41-908e-db02f1c508f9/volumes"
Oct 30 17:33:56.054448 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:56.054420    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3260f8-pws4g"
Oct 30 17:33:56.163546 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:56.163508    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0-maas-api-openshift-ca\") pod \"3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0\" (UID: \"3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0\") "
Oct 30 17:33:56.163742 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:56.163675    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0-kube-api-access-pw8jw\" (UniqueName: \"kubernetes.io/projected/3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0-kube-api-access-pw8jw\") pod \"3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0\" (UID: \"3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0\") "
Oct 30 17:33:56.163795 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:56.163785    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0-maas-api-xks-ca\") pod \"3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0\" (UID: \"3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0\") "
Oct 30 17:33:56.163962 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:56.163900    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0-maas-api-openshift-ca" pod "3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0" (UID: "3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:33:56.164440 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:56.164417    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0-maas-api-openshift-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:56.166374 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:56.166348    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0-maas-api-xks-ca" pod "3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0" (UID: "3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:33:56.166374 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:56.166366    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0-kube-api-access-pw8jw" pod "3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0" (UID: "3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0"). InnerVolumeSpecName "kube-api-access-pw8jw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:33:56.265724 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:56.265645    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw8jw\" (UniqueName: \"kubernetes.io/projected/3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0-kube-api-access-pw8jw\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:56.265724 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:56.265673    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0-maas-api-xks-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:33:56.450482 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:56.450445    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq"]
Oct 30 17:33:56.903880 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:56.903833    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3260f8-pws4g" event={"ID":"3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0","Type":"ContainerDied","Data":"b023365a98a3f5298f51cce300589c51d4ab0f0d1c18b6ee10524042a5537e58"}
Oct 30 17:33:56.903880 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:56.903879    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3260f8-pws4g"
Oct 30 17:33:56.904079 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:56.903901    4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b023365a98a3f5298f51cce300589c51d4ab0f0d1c18b6ee10524042a5537e58"
Oct 30 17:33:56.979527 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:33:56.979487    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:33:58.117466 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:58.117429    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33075c97-7mc6t"]
Oct 30 17:33:58.118383 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:58.118357    4433 cpu_manager.go:414] "containerMap: removing container" podUID="3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0" containerName="revoke-keys"
Oct 30 17:33:58.118511 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:58.118386    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0" containerName="revoke-keys"
Oct 30 17:33:58.118511 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:58.118451    4433 cpu_manager.go:414] "containerMap: removing container" podUID="1edce449-0010-4d41-908e-db02f1c508f9" containerName="istio-proxy"
Oct 30 17:33:58.118511 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:58.118462    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1edce449-0010-4d41-908e-db02f1c508f9" containerName="istio-proxy"
Oct 30 17:33:58.118681 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:58.118666    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="1edce449-0010-4d41-908e-db02f1c508f9" containerName="istio-proxy"
Oct 30 17:33:58.118717 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:58.118698    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0" containerName="revoke-keys"
Oct 30 17:33:58.123085 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:58.123062    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33075c97-7mc6t"
Oct 30 17:33:58.142246 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:58.142213    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33075c97-7mc6t"]
Oct 30 17:33:58.208381 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:58.208345    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlfkj\" (UniqueName: \"kubernetes.io/projected/35956888-3e58-4794-8613-bf44931f5129-kube-api-access-rlfkj\") pod \"maas-api-revoke-sub-33075c97-7mc6t\" (UID: \"35956888-3e58-4794-8613-bf44931f5129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33075c97-7mc6t"
Oct 30 17:33:58.208553 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:58.208393    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35956888-3e58-4794-8613-bf44931f5129-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33075c97-7mc6t\" (UID: \"35956888-3e58-4794-8613-bf44931f5129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33075c97-7mc6t"
Oct 30 17:33:58.208611 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:58.208579    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35956888-3e58-4794-8613-bf44931f5129-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33075c97-7mc6t\" (UID: \"35956888-3e58-4794-8613-bf44931f5129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33075c97-7mc6t"
Oct 30 17:33:58.216597 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:58.216574    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:33:58.216597 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:58.216574    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\""
Oct 30 17:33:58.310150 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:58.310102    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlfkj\" (UniqueName: \"kubernetes.io/projected/35956888-3e58-4794-8613-bf44931f5129-kube-api-access-rlfkj\") pod \"maas-api-revoke-sub-33075c97-7mc6t\" (UID: \"35956888-3e58-4794-8613-bf44931f5129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33075c97-7mc6t"
Oct 30 17:33:58.310350 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:58.310156    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35956888-3e58-4794-8613-bf44931f5129-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33075c97-7mc6t\" (UID: \"35956888-3e58-4794-8613-bf44931f5129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33075c97-7mc6t"
Oct 30 17:33:58.310477 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:58.310360    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35956888-3e58-4794-8613-bf44931f5129-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33075c97-7mc6t\" (UID: \"35956888-3e58-4794-8613-bf44931f5129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33075c97-7mc6t"
Oct 30 17:33:58.310997 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:58.310974    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35956888-3e58-4794-8613-bf44931f5129-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33075c97-7mc6t\" (UID: \"35956888-3e58-4794-8613-bf44931f5129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33075c97-7mc6t"
Oct 30 17:33:58.312699 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:58.312665    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35956888-3e58-4794-8613-bf44931f5129-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33075c97-7mc6t\" (UID: \"35956888-3e58-4794-8613-bf44931f5129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33075c97-7mc6t"
Oct 30 17:33:58.332892 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:58.332858    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlfkj\" (UniqueName: \"kubernetes.io/projected/35956888-3e58-4794-8613-bf44931f5129-kube-api-access-rlfkj\") pod \"maas-api-revoke-sub-33075c97-7mc6t\" (UID: \"35956888-3e58-4794-8613-bf44931f5129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33075c97-7mc6t"
Oct 30 17:33:58.446566 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:58.446534    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33075c97-7mc6t"
Oct 30 17:33:58.816972 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:58.816944    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33075c97-7mc6t"]
Oct 30 17:33:58.818903 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:33:58.818875    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35956888_3e58_4794_8613_bf44931f5129.slice/crio-07e53957d0b6bd88b51d70cff7052b00213fd8eb4aac8795d10f9601c613cfd9 WatchSource:0}: Error finding container 07e53957d0b6bd88b51d70cff7052b00213fd8eb4aac8795d10f9601c613cfd9: Status 404 returned error can't find the container with id 07e53957d0b6bd88b51d70cff7052b00213fd8eb4aac8795d10f9601c613cfd9
Oct 30 17:33:58.918457 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:58.918389    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33075c97-7mc6t" event={"ID":"35956888-3e58-4794-8613-bf44931f5129","Type":"ContainerStarted","Data":"07e53957d0b6bd88b51d70cff7052b00213fd8eb4aac8795d10f9601c613cfd9"}
Oct 30 17:33:59.926647 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:59.926610    4433 generic.go:356] "Generic (PLEG): container finished" podID="35956888-3e58-4794-8613-bf44931f5129" containerID="780ea53cd9efd134d85dd4c700479f441e5930a41885888456d58e225cc30d64" exitCode=0
Oct 30 17:33:59.927046 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:33:59.926700    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33075c97-7mc6t" event={"ID":"35956888-3e58-4794-8613-bf44931f5129","Type":"ContainerDied","Data":"780ea53cd9efd134d85dd4c700479f441e5930a41885888456d58e225cc30d64"}
Oct 30 17:34:01.105126 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:01.105100    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33075c97-7mc6t"
Oct 30 17:34:01.149931 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:01.149885    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35956888-3e58-4794-8613-bf44931f5129-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35956888-3e58-4794-8613-bf44931f5129-maas-api-xks-ca\") pod \"35956888-3e58-4794-8613-bf44931f5129\" (UID: \"35956888-3e58-4794-8613-bf44931f5129\") "
Oct 30 17:34:01.150145 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:01.149960    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35956888-3e58-4794-8613-bf44931f5129-kube-api-access-rlfkj\" (UniqueName: \"kubernetes.io/projected/35956888-3e58-4794-8613-bf44931f5129-kube-api-access-rlfkj\") pod \"35956888-3e58-4794-8613-bf44931f5129\" (UID: \"35956888-3e58-4794-8613-bf44931f5129\") "
Oct 30 17:34:01.150252 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:01.150233    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35956888-3e58-4794-8613-bf44931f5129-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35956888-3e58-4794-8613-bf44931f5129-maas-api-openshift-ca\") pod \"35956888-3e58-4794-8613-bf44931f5129\" (UID: \"35956888-3e58-4794-8613-bf44931f5129\") "
Oct 30 17:34:01.151061 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:01.151031    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35956888-3e58-4794-8613-bf44931f5129-maas-api-openshift-ca" pod "35956888-3e58-4794-8613-bf44931f5129" (UID: "35956888-3e58-4794-8613-bf44931f5129"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:34:01.154934 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:01.153544    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35956888-3e58-4794-8613-bf44931f5129-maas-api-xks-ca" pod "35956888-3e58-4794-8613-bf44931f5129" (UID: "35956888-3e58-4794-8613-bf44931f5129"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:34:01.154934 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:01.153578    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35956888-3e58-4794-8613-bf44931f5129-kube-api-access-rlfkj" pod "35956888-3e58-4794-8613-bf44931f5129" (UID: "35956888-3e58-4794-8613-bf44931f5129"). InnerVolumeSpecName "kube-api-access-rlfkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:34:01.252092 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:01.252009    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35956888-3e58-4794-8613-bf44931f5129-maas-api-openshift-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:34:01.252092 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:01.252041    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35956888-3e58-4794-8613-bf44931f5129-maas-api-xks-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:34:01.252092 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:01.252051    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlfkj\" (UniqueName: \"kubernetes.io/projected/35956888-3e58-4794-8613-bf44931f5129-kube-api-access-rlfkj\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:34:01.941974 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:01.941939    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33075c97-7mc6t"
Oct 30 17:34:01.941974 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:01.941965    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33075c97-7mc6t" event={"ID":"35956888-3e58-4794-8613-bf44931f5129","Type":"ContainerDied","Data":"07e53957d0b6bd88b51d70cff7052b00213fd8eb4aac8795d10f9601c613cfd9"}
Oct 30 17:34:01.942215 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:01.942020    4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07e53957d0b6bd88b51d70cff7052b00213fd8eb4aac8795d10f9601c613cfd9"
Oct 30 17:34:06.922639 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:06.922600    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a949e7-7cff8d95bb-9r6gv"]
Oct 30 17:34:11.480851 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:11.480781    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq" podUID="ba03c7a7-48e4-4051-91d4-9a030790fa8a" containerName="main" containerID="cri-o://169568da9f2606562778832bc83a39b45d409e21d589a2bdbf5c58be348b3c54" gracePeriod=45
Oct 30 17:34:11.783021 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:11.782996    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq"
Oct 30 17:34:11.902445 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:11.902388    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba03c7a7-48e4-4051-91d4-9a030790fa8a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba03c7a7-48e4-4051-91d4-9a030790fa8a-tmp-dir\") pod \"ba03c7a7-48e4-4051-91d4-9a030790fa8a\" (UID: \"ba03c7a7-48e4-4051-91d4-9a030790fa8a\") "
Oct 30 17:34:11.902621 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:11.902475    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba03c7a7-48e4-4051-91d4-9a030790fa8a-tls-certs\" (UniqueName: \"kubernetes.io/secret/ba03c7a7-48e4-4051-91d4-9a030790fa8a-tls-certs\") pod \"ba03c7a7-48e4-4051-91d4-9a030790fa8a\" (UID: \"ba03c7a7-48e4-4051-91d4-9a030790fa8a\") "
Oct 30 17:34:11.902621 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:11.902519    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba03c7a7-48e4-4051-91d4-9a030790fa8a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba03c7a7-48e4-4051-91d4-9a030790fa8a-dshm\") pod \"ba03c7a7-48e4-4051-91d4-9a030790fa8a\" (UID: \"ba03c7a7-48e4-4051-91d4-9a030790fa8a\") "
Oct 30 17:34:11.902708 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:11.902671    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba03c7a7-48e4-4051-91d4-9a030790fa8a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba03c7a7-48e4-4051-91d4-9a030790fa8a-model-cache\") pod \"ba03c7a7-48e4-4051-91d4-9a030790fa8a\" (UID: \"ba03c7a7-48e4-4051-91d4-9a030790fa8a\") "
Oct 30 17:34:11.902708 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:11.902685    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba03c7a7-48e4-4051-91d4-9a030790fa8a-tmp-dir" pod "ba03c7a7-48e4-4051-91d4-9a030790fa8a" (UID: "ba03c7a7-48e4-4051-91d4-9a030790fa8a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:34:11.902855 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:11.902836    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba03c7a7-48e4-4051-91d4-9a030790fa8a-home\" (UniqueName: \"kubernetes.io/empty-dir/ba03c7a7-48e4-4051-91d4-9a030790fa8a-home\") pod \"ba03c7a7-48e4-4051-91d4-9a030790fa8a\" (UID: \"ba03c7a7-48e4-4051-91d4-9a030790fa8a\") "
Oct 30 17:34:11.902918 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:11.902907    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba03c7a7-48e4-4051-91d4-9a030790fa8a-kube-api-access-2cvcs\" (UniqueName: \"kubernetes.io/projected/ba03c7a7-48e4-4051-91d4-9a030790fa8a-kube-api-access-2cvcs\") pod \"ba03c7a7-48e4-4051-91d4-9a030790fa8a\" (UID: \"ba03c7a7-48e4-4051-91d4-9a030790fa8a\") "
Oct 30 17:34:11.902974 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:11.902928    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba03c7a7-48e4-4051-91d4-9a030790fa8a-model-cache" pod "ba03c7a7-48e4-4051-91d4-9a030790fa8a" (UID: "ba03c7a7-48e4-4051-91d4-9a030790fa8a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:34:11.903084 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:11.903064    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba03c7a7-48e4-4051-91d4-9a030790fa8a-home" pod "ba03c7a7-48e4-4051-91d4-9a030790fa8a" (UID: "ba03c7a7-48e4-4051-91d4-9a030790fa8a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:34:11.903522 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:11.903488    4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba03c7a7-48e4-4051-91d4-9a030790fa8a-home\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:34:11.903522 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:11.903514    4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba03c7a7-48e4-4051-91d4-9a030790fa8a-tmp-dir\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:34:11.903522 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:11.903525    4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba03c7a7-48e4-4051-91d4-9a030790fa8a-model-cache\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:34:11.905016 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:11.904977    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba03c7a7-48e4-4051-91d4-9a030790fa8a-dshm" pod "ba03c7a7-48e4-4051-91d4-9a030790fa8a" (UID: "ba03c7a7-48e4-4051-91d4-9a030790fa8a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:34:11.905164 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:11.905146    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba03c7a7-48e4-4051-91d4-9a030790fa8a-kube-api-access-2cvcs" pod "ba03c7a7-48e4-4051-91d4-9a030790fa8a" (UID: "ba03c7a7-48e4-4051-91d4-9a030790fa8a"). InnerVolumeSpecName "kube-api-access-2cvcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:34:11.905264 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:11.905244    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba03c7a7-48e4-4051-91d4-9a030790fa8a-tls-certs" pod "ba03c7a7-48e4-4051-91d4-9a030790fa8a" (UID: "ba03c7a7-48e4-4051-91d4-9a030790fa8a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:34:11.924334 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:11.924303    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-7cff8d95bb-9r6gv" podUID="db411383-a64b-4614-b496-abe53f6cc5c0" containerName="payload-processing" containerID="cri-o://48c21080dfcbc80ed5be0caa5bbb0ca19efe4abdbcb32d66a9e06011c8c48f50" gracePeriod=25
Oct 30 17:34:12.005281 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:12.005188    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cvcs\" (UniqueName: \"kubernetes.io/projected/ba03c7a7-48e4-4051-91d4-9a030790fa8a-kube-api-access-2cvcs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:34:12.005281 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:12.005221    4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba03c7a7-48e4-4051-91d4-9a030790fa8a-tls-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:34:12.005281 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:12.005237    4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba03c7a7-48e4-4051-91d4-9a030790fa8a-dshm\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:34:12.016952 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:12.016917    4433 generic.go:356] "Generic (PLEG): container finished" podID="ba03c7a7-48e4-4051-91d4-9a030790fa8a" containerID="169568da9f2606562778832bc83a39b45d409e21d589a2bdbf5c58be348b3c54" exitCode=0
Oct 30 17:34:12.017132 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:12.016957    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq" event={"ID":"ba03c7a7-48e4-4051-91d4-9a030790fa8a","Type":"ContainerDied","Data":"169568da9f2606562778832bc83a39b45d409e21d589a2bdbf5c58be348b3c54"}
Oct 30 17:34:12.017132 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:12.016980    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq"
Oct 30 17:34:12.017132 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:12.016991    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq" event={"ID":"ba03c7a7-48e4-4051-91d4-9a030790fa8a","Type":"ContainerDied","Data":"4a65f9cf9c102bc2562ca801f3a854b7e5732900de8f41e5fb23e68f58f091b4"}
Oct 30 17:34:12.017132 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:12.017012    4433 scope.go:122] "RemoveContainer" containerID="169568da9f2606562778832bc83a39b45d409e21d589a2bdbf5c58be348b3c54"
Oct 30 17:34:12.037724 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:12.037704    4433 scope.go:122] "RemoveContainer" containerID="169568da9f2606562778832bc83a39b45d409e21d589a2bdbf5c58be348b3c54"
Oct 30 17:34:12.038003 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:34:12.037983    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"169568da9f2606562778832bc83a39b45d409e21d589a2bdbf5c58be348b3c54\": container with ID starting with 169568da9f2606562778832bc83a39b45d409e21d589a2bdbf5c58be348b3c54 not found: ID does not exist" containerID="169568da9f2606562778832bc83a39b45d409e21d589a2bdbf5c58be348b3c54"
Oct 30 17:34:12.038059 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:12.038017    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"169568da9f2606562778832bc83a39b45d409e21d589a2bdbf5c58be348b3c54"} err="failed to get container status \"169568da9f2606562778832bc83a39b45d409e21d589a2bdbf5c58be348b3c54\": rpc error: code = NotFound desc = could not find container \"169568da9f2606562778832bc83a39b45d409e21d589a2bdbf5c58be348b3c54\": container with ID starting with 169568da9f2606562778832bc83a39b45d409e21d589a2bdbf5c58be348b3c54 not found: ID does not exist"
Oct 30 17:34:12.069696 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:12.069659    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq"]
Oct 30 17:34:12.074243 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:12.074213    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq"]
Oct 30 17:34:12.183786 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:12.183758    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-7cff8d95bb-9r6gv"
Oct 30 17:34:12.310382 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:12.310298    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db411383-a64b-4614-b496-abe53f6cc5c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db411383-a64b-4614-b496-abe53f6cc5c0-plugins-config-volume\") pod \"db411383-a64b-4614-b496-abe53f6cc5c0\" (UID: \"db411383-a64b-4614-b496-abe53f6cc5c0\") "
Oct 30 17:34:12.310572 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:12.310435    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db411383-a64b-4614-b496-abe53f6cc5c0-kube-api-access-7894b\" (UniqueName: \"kubernetes.io/projected/db411383-a64b-4614-b496-abe53f6cc5c0-kube-api-access-7894b\") pod \"db411383-a64b-4614-b496-abe53f6cc5c0\" (UID: \"db411383-a64b-4614-b496-abe53f6cc5c0\") "
Oct 30 17:34:12.310774 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:12.310746    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db411383-a64b-4614-b496-abe53f6cc5c0-plugins-config-volume" pod "db411383-a64b-4614-b496-abe53f6cc5c0" (UID: "db411383-a64b-4614-b496-abe53f6cc5c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:34:12.311016 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:12.310996    4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db411383-a64b-4614-b496-abe53f6cc5c0-plugins-config-volume\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:34:12.312654 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:12.312625    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db411383-a64b-4614-b496-abe53f6cc5c0-kube-api-access-7894b" pod "db411383-a64b-4614-b496-abe53f6cc5c0" (UID: "db411383-a64b-4614-b496-abe53f6cc5c0"). InnerVolumeSpecName "kube-api-access-7894b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:34:12.412251 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:12.412218    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7894b\" (UniqueName: \"kubernetes.io/projected/db411383-a64b-4614-b496-abe53f6cc5c0-kube-api-access-7894b\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:34:12.669999 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:12.669906    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-b87f3c/auth-test-model-b87f3c-kserve-6c6d6fb985-5lwwq" podUID="ba03c7a7-48e4-4051-91d4-9a030790fa8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.168:8000/ready\": context deadline exceeded"
Oct 30 17:34:13.026273 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:13.026231    4433 generic.go:356] "Generic (PLEG): container finished" podID="db411383-a64b-4614-b496-abe53f6cc5c0" containerID="48c21080dfcbc80ed5be0caa5bbb0ca19efe4abdbcb32d66a9e06011c8c48f50" exitCode=0
Oct 30 17:34:13.026471 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:13.026298    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-7cff8d95bb-9r6gv"
Oct 30 17:34:13.026471 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:13.026314    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-7cff8d95bb-9r6gv" event={"ID":"db411383-a64b-4614-b496-abe53f6cc5c0","Type":"ContainerDied","Data":"48c21080dfcbc80ed5be0caa5bbb0ca19efe4abdbcb32d66a9e06011c8c48f50"}
Oct 30 17:34:13.026471 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:13.026374    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-7cff8d95bb-9r6gv" event={"ID":"db411383-a64b-4614-b496-abe53f6cc5c0","Type":"ContainerDied","Data":"0a85a2326d94fa0caac06c6f1c9c944fe5b1a2da7ec071728c5d73ae9da07c09"}
Oct 30 17:34:13.026471 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:13.026398    4433 scope.go:122] "RemoveContainer" containerID="48c21080dfcbc80ed5be0caa5bbb0ca19efe4abdbcb32d66a9e06011c8c48f50"
Oct 30 17:34:13.037843 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:13.037824    4433 scope.go:122] "RemoveContainer" containerID="48c21080dfcbc80ed5be0caa5bbb0ca19efe4abdbcb32d66a9e06011c8c48f50"
Oct 30 17:34:13.038173 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:34:13.038153    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48c21080dfcbc80ed5be0caa5bbb0ca19efe4abdbcb32d66a9e06011c8c48f50\": container with ID starting with 48c21080dfcbc80ed5be0caa5bbb0ca19efe4abdbcb32d66a9e06011c8c48f50 not found: ID does not exist" containerID="48c21080dfcbc80ed5be0caa5bbb0ca19efe4abdbcb32d66a9e06011c8c48f50"
Oct 30 17:34:13.038224 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:13.038183    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48c21080dfcbc80ed5be0caa5bbb0ca19efe4abdbcb32d66a9e06011c8c48f50"} err="failed to get container status \"48c21080dfcbc80ed5be0caa5bbb0ca19efe4abdbcb32d66a9e06011c8c48f50\": rpc error: code = NotFound desc = could not find container \"48c21080dfcbc80ed5be0caa5bbb0ca19efe4abdbcb32d66a9e06011c8c48f50\": container with ID starting with 48c21080dfcbc80ed5be0caa5bbb0ca19efe4abdbcb32d66a9e06011c8c48f50 not found: ID does not exist"
Oct 30 17:34:13.078157 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:13.078121    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a949e7-7cff8d95bb-9r6gv"]
Oct 30 17:34:13.089237 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:13.089204    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a949e7-7cff8d95bb-9r6gv"]
Oct 30 17:34:13.767141 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:13.765794    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba03c7a7-48e4-4051-91d4-9a030790fa8a" path="/var/lib/kubelet/pods/ba03c7a7-48e4-4051-91d4-9a030790fa8a/volumes"
Oct 30 17:34:13.770276 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:13.768045    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db411383-a64b-4614-b496-abe53f6cc5c0" path="/var/lib/kubelet/pods/db411383-a64b-4614-b496-abe53f6cc5c0/volumes"
Oct 30 17:34:17.038776 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:17.038700    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:34:27.091022 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:34:27.090983    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:34:42.650325 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:42.650287    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3260f8-pws4g"]
Oct 30 17:34:42.662623 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:42.662586    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3260f8-pws4g"]
Oct 30 17:34:43.696736 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:43.696700    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0" path="/var/lib/kubelet/pods/3c1cc3f6-c25c-4c55-b8f8-76fa94b25aa0/volumes"
Oct 30 17:34:44.958545 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:44.958510    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7b9df-2lphz"]
Oct 30 17:34:44.969116 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:44.969090    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7b9df-2lphz"]
Oct 30 17:34:45.696315 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:45.696280    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3199ec19-a07d-4225-a9d8-1926f089ae6b" path="/var/lib/kubelet/pods/3199ec19-a07d-4225-a9d8-1926f089ae6b/volumes"
Oct 30 17:34:50.112266 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:50.112231    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"]
Oct 30 17:34:50.113044 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:50.113012    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch" podUID="0cd29788-1dd4-4f3a-910e-0c180daed229" containerName="istio-proxy" containerID="cri-o://87903d0308a7de56c43842895dc53610793ff799db9e95c55330d904c978c715" gracePeriod=30
Oct 30 17:34:53.196618 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:53.196572    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"]
Oct 30 17:34:53.198161 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:53.197801    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp" podUID="946311cc-f655-4798-bce5-10ad7d73c9e6" containerName="istio-proxy" containerID="cri-o://f2cbcd6bafafd3e966f436bf1923a9997029bcb4c93a7058cc01ee3e463ab3b4" gracePeriod=30
Oct 30 17:34:53.209479 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:53.209442    4433 patch_prober.go:29] interesting pod/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:34:53.209723 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:53.209689    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch" podUID="0cd29788-1dd4-4f3a-910e-0c180daed229" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:34:55.427450 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.427418    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:34:55.485520 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.485487    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd29788-1dd4-4f3a-910e-0c180daed229-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd29788-1dd4-4f3a-910e-0c180daed229-workload-socket\") pod \"0cd29788-1dd4-4f3a-910e-0c180daed229\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") "
Oct 30 17:34:55.485719 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.485625    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cd29788-1dd4-4f3a-910e-0c180daed229-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cd29788-1dd4-4f3a-910e-0c180daed229-istiod-ca-cert\") pod \"0cd29788-1dd4-4f3a-910e-0c180daed229\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") "
Oct 30 17:34:55.485792 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.485718    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cd29788-1dd4-4f3a-910e-0c180daed229-istio-token\" (UniqueName: \"kubernetes.io/projected/0cd29788-1dd4-4f3a-910e-0c180daed229-istio-token\") pod \"0cd29788-1dd4-4f3a-910e-0c180daed229\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") "
Oct 30 17:34:55.485851 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.485804    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd29788-1dd4-4f3a-910e-0c180daed229-workload-socket" pod "0cd29788-1dd4-4f3a-910e-0c180daed229" (UID: "0cd29788-1dd4-4f3a-910e-0c180daed229"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:34:55.485851 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.485821    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0cd29788-1dd4-4f3a-910e-0c180daed229-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cd29788-1dd4-4f3a-910e-0c180daed229-istio-podinfo\") pod \"0cd29788-1dd4-4f3a-910e-0c180daed229\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") "
Oct 30 17:34:55.485961 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.485906    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd29788-1dd4-4f3a-910e-0c180daed229-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cd29788-1dd4-4f3a-910e-0c180daed229-workload-certs\") pod \"0cd29788-1dd4-4f3a-910e-0c180daed229\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") "
Oct 30 17:34:55.486019 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.485998    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd29788-1dd4-4f3a-910e-0c180daed229-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cd29788-1dd4-4f3a-910e-0c180daed229-istio-data\") pod \"0cd29788-1dd4-4f3a-910e-0c180daed229\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") "
Oct 30 17:34:55.486079 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.486032    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cd29788-1dd4-4f3a-910e-0c180daed229-kube-api-access-vrnf2\" (UniqueName: \"kubernetes.io/projected/0cd29788-1dd4-4f3a-910e-0c180daed229-kube-api-access-vrnf2\") pod \"0cd29788-1dd4-4f3a-910e-0c180daed229\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") "
Oct 30 17:34:55.486079 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.486062    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd29788-1dd4-4f3a-910e-0c180daed229-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cd29788-1dd4-4f3a-910e-0c180daed229-istio-envoy\") pod \"0cd29788-1dd4-4f3a-910e-0c180daed229\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") "
Oct 30 17:34:55.486079 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.486062    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cd29788-1dd4-4f3a-910e-0c180daed229-istiod-ca-cert" pod "0cd29788-1dd4-4f3a-910e-0c180daed229" (UID: "0cd29788-1dd4-4f3a-910e-0c180daed229"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:34:55.486271 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.486249    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd29788-1dd4-4f3a-910e-0c180daed229-workload-certs" pod "0cd29788-1dd4-4f3a-910e-0c180daed229" (UID: "0cd29788-1dd4-4f3a-910e-0c180daed229"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:34:55.486336 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.486131    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd29788-1dd4-4f3a-910e-0c180daed229-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd29788-1dd4-4f3a-910e-0c180daed229-credential-socket\") pod \"0cd29788-1dd4-4f3a-910e-0c180daed229\" (UID: \"0cd29788-1dd4-4f3a-910e-0c180daed229\") "
Oct 30 17:34:55.486336 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.486284    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd29788-1dd4-4f3a-910e-0c180daed229-istio-data" pod "0cd29788-1dd4-4f3a-910e-0c180daed229" (UID: "0cd29788-1dd4-4f3a-910e-0c180daed229"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:34:55.486517 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.486493    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd29788-1dd4-4f3a-910e-0c180daed229-credential-socket" pod "0cd29788-1dd4-4f3a-910e-0c180daed229" (UID: "0cd29788-1dd4-4f3a-910e-0c180daed229"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:34:55.486978 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.486960    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd29788-1dd4-4f3a-910e-0c180daed229-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:34:55.487082 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.486982    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cd29788-1dd4-4f3a-910e-0c180daed229-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:34:55.487082 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.486998    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cd29788-1dd4-4f3a-910e-0c180daed229-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:34:55.487082 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.487009    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cd29788-1dd4-4f3a-910e-0c180daed229-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:34:55.487082 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.487021    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd29788-1dd4-4f3a-910e-0c180daed229-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:34:55.488328 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.488286    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cd29788-1dd4-4f3a-910e-0c180daed229-istio-podinfo" pod "0cd29788-1dd4-4f3a-910e-0c180daed229" (UID: "0cd29788-1dd4-4f3a-910e-0c180daed229"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:34:55.488712 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.488681    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd29788-1dd4-4f3a-910e-0c180daed229-istio-envoy" pod "0cd29788-1dd4-4f3a-910e-0c180daed229" (UID: "0cd29788-1dd4-4f3a-910e-0c180daed229"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:34:55.488823 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.488761    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd29788-1dd4-4f3a-910e-0c180daed229-istio-token" pod "0cd29788-1dd4-4f3a-910e-0c180daed229" (UID: "0cd29788-1dd4-4f3a-910e-0c180daed229"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:34:55.488993 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.488963    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd29788-1dd4-4f3a-910e-0c180daed229-kube-api-access-vrnf2" pod "0cd29788-1dd4-4f3a-910e-0c180daed229" (UID: "0cd29788-1dd4-4f3a-910e-0c180daed229"). InnerVolumeSpecName "kube-api-access-vrnf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:34:55.588908 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.588873    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cd29788-1dd4-4f3a-910e-0c180daed229-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:34:55.588908 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.588904    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cd29788-1dd4-4f3a-910e-0c180daed229-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:34:55.588908 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.588913    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrnf2\" (UniqueName: \"kubernetes.io/projected/0cd29788-1dd4-4f3a-910e-0c180daed229-kube-api-access-vrnf2\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:34:55.589164 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.588922    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cd29788-1dd4-4f3a-910e-0c180daed229-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:34:55.658319 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.658229    4433 generic.go:356] "Generic (PLEG): container finished" podID="0cd29788-1dd4-4f3a-910e-0c180daed229" containerID="87903d0308a7de56c43842895dc53610793ff799db9e95c55330d904c978c715" exitCode=0
Oct 30 17:34:55.658319 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.658281    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"
Oct 30 17:34:55.658319 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.658306    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch" event={"ID":"0cd29788-1dd4-4f3a-910e-0c180daed229","Type":"ContainerDied","Data":"87903d0308a7de56c43842895dc53610793ff799db9e95c55330d904c978c715"}
Oct 30 17:34:55.658610 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.658350    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch" event={"ID":"0cd29788-1dd4-4f3a-910e-0c180daed229","Type":"ContainerDied","Data":"fb76cfdd30c9f41f6ead08b2c962f876b0d89d4385d5c94854490d9d29c04fe7"}
Oct 30 17:34:55.658610 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.658371    4433 scope.go:122] "RemoveContainer" containerID="87903d0308a7de56c43842895dc53610793ff799db9e95c55330d904c978c715"
Oct 30 17:34:55.669425 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.669393    4433 scope.go:122] "RemoveContainer" containerID="7c3a2a585b69006a59fa9b1fdcd7a0595f115e43433b23393e25ce2809c76c38"
Oct 30 17:34:55.682730 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.682706    4433 scope.go:122] "RemoveContainer" containerID="87903d0308a7de56c43842895dc53610793ff799db9e95c55330d904c978c715"
Oct 30 17:34:55.683026 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:34:55.683005    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87903d0308a7de56c43842895dc53610793ff799db9e95c55330d904c978c715\": container with ID starting with 87903d0308a7de56c43842895dc53610793ff799db9e95c55330d904c978c715 not found: ID does not exist" containerID="87903d0308a7de56c43842895dc53610793ff799db9e95c55330d904c978c715"
Oct 30 17:34:55.683079 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.683039    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87903d0308a7de56c43842895dc53610793ff799db9e95c55330d904c978c715"} err="failed to get container status \"87903d0308a7de56c43842895dc53610793ff799db9e95c55330d904c978c715\": rpc error: code = NotFound desc = could not find container \"87903d0308a7de56c43842895dc53610793ff799db9e95c55330d904c978c715\": container with ID starting with 87903d0308a7de56c43842895dc53610793ff799db9e95c55330d904c978c715 not found: ID does not exist"
Oct 30 17:34:55.683079 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.683057    4433 scope.go:122] "RemoveContainer" containerID="7c3a2a585b69006a59fa9b1fdcd7a0595f115e43433b23393e25ce2809c76c38"
Oct 30 17:34:55.683334 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:34:55.683310    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c3a2a585b69006a59fa9b1fdcd7a0595f115e43433b23393e25ce2809c76c38\": container with ID starting with 7c3a2a585b69006a59fa9b1fdcd7a0595f115e43433b23393e25ce2809c76c38 not found: ID does not exist" containerID="7c3a2a585b69006a59fa9b1fdcd7a0595f115e43433b23393e25ce2809c76c38"
Oct 30 17:34:55.683418 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.683344    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c3a2a585b69006a59fa9b1fdcd7a0595f115e43433b23393e25ce2809c76c38"} err="failed to get container status \"7c3a2a585b69006a59fa9b1fdcd7a0595f115e43433b23393e25ce2809c76c38\": rpc error: code = NotFound desc = could not find container \"7c3a2a585b69006a59fa9b1fdcd7a0595f115e43433b23393e25ce2809c76c38\": container with ID starting with 7c3a2a585b69006a59fa9b1fdcd7a0595f115e43433b23393e25ce2809c76c38 not found: ID does not exist"
Oct 30 17:34:55.710784 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.710759    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"]
Oct 30 17:34:55.716436 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:55.716395    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-3260f8-openshift-default-5d86b7b67f-kdvch"]
Oct 30 17:34:56.977930 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:34:56.977892    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:34:57.727825 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:57.727787    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cd29788-1dd4-4f3a-910e-0c180daed229" path="/var/lib/kubelet/pods/0cd29788-1dd4-4f3a-910e-0c180daed229/volumes"
Oct 30 17:34:58.485652 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.485584    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"
Oct 30 17:34:58.637791 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.637759    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/946311cc-f655-4798-bce5-10ad7d73c9e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/946311cc-f655-4798-bce5-10ad7d73c9e6-credential-socket\") pod \"946311cc-f655-4798-bce5-10ad7d73c9e6\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") "
Oct 30 17:34:58.637979 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.637823    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/946311cc-f655-4798-bce5-10ad7d73c9e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/946311cc-f655-4798-bce5-10ad7d73c9e6-istiod-ca-cert\") pod \"946311cc-f655-4798-bce5-10ad7d73c9e6\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") "
Oct 30 17:34:58.637979 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.637844    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/946311cc-f655-4798-bce5-10ad7d73c9e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/946311cc-f655-4798-bce5-10ad7d73c9e6-istio-envoy\") pod \"946311cc-f655-4798-bce5-10ad7d73c9e6\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") "
Oct 30 17:34:58.637979 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.637883    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/946311cc-f655-4798-bce5-10ad7d73c9e6-kube-api-access-n4sdd\" (UniqueName: \"kubernetes.io/projected/946311cc-f655-4798-bce5-10ad7d73c9e6-kube-api-access-n4sdd\") pod \"946311cc-f655-4798-bce5-10ad7d73c9e6\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") "
Oct 30 17:34:58.637979 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.637908    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/946311cc-f655-4798-bce5-10ad7d73c9e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/946311cc-f655-4798-bce5-10ad7d73c9e6-istio-data\") pod \"946311cc-f655-4798-bce5-10ad7d73c9e6\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") "
Oct 30 17:34:58.638210 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.638033    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/946311cc-f655-4798-bce5-10ad7d73c9e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/946311cc-f655-4798-bce5-10ad7d73c9e6-istio-podinfo\") pod \"946311cc-f655-4798-bce5-10ad7d73c9e6\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") "
Oct 30 17:34:58.638210 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.638052    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/946311cc-f655-4798-bce5-10ad7d73c9e6-credential-socket" pod "946311cc-f655-4798-bce5-10ad7d73c9e6" (UID: "946311cc-f655-4798-bce5-10ad7d73c9e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:34:58.638210 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.638077    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/946311cc-f655-4798-bce5-10ad7d73c9e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/946311cc-f655-4798-bce5-10ad7d73c9e6-workload-socket\") pod \"946311cc-f655-4798-bce5-10ad7d73c9e6\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") "
Oct 30 17:34:58.638210 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.638196    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/946311cc-f655-4798-bce5-10ad7d73c9e6-istio-data" pod "946311cc-f655-4798-bce5-10ad7d73c9e6" (UID: "946311cc-f655-4798-bce5-10ad7d73c9e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:34:58.638562 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.638231    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/946311cc-f655-4798-bce5-10ad7d73c9e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/946311cc-f655-4798-bce5-10ad7d73c9e6-workload-certs\") pod \"946311cc-f655-4798-bce5-10ad7d73c9e6\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") "
Oct 30 17:34:58.638562 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.638313    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/946311cc-f655-4798-bce5-10ad7d73c9e6-istio-token\" (UniqueName: \"kubernetes.io/projected/946311cc-f655-4798-bce5-10ad7d73c9e6-istio-token\") pod \"946311cc-f655-4798-bce5-10ad7d73c9e6\" (UID: \"946311cc-f655-4798-bce5-10ad7d73c9e6\") "
Oct 30 17:34:58.638562 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.638311    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/946311cc-f655-4798-bce5-10ad7d73c9e6-istiod-ca-cert" pod "946311cc-f655-4798-bce5-10ad7d73c9e6" (UID: "946311cc-f655-4798-bce5-10ad7d73c9e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:34:58.638562 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.638340    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/946311cc-f655-4798-bce5-10ad7d73c9e6-workload-socket" pod "946311cc-f655-4798-bce5-10ad7d73c9e6" (UID: "946311cc-f655-4798-bce5-10ad7d73c9e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:34:58.638562 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.638535    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/946311cc-f655-4798-bce5-10ad7d73c9e6-workload-certs" pod "946311cc-f655-4798-bce5-10ad7d73c9e6" (UID: "946311cc-f655-4798-bce5-10ad7d73c9e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:34:58.639017 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.638989    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/946311cc-f655-4798-bce5-10ad7d73c9e6-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:34:58.639103 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.639022    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/946311cc-f655-4798-bce5-10ad7d73c9e6-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:34:58.639103 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.639038    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/946311cc-f655-4798-bce5-10ad7d73c9e6-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:34:58.639103 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.639051    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/946311cc-f655-4798-bce5-10ad7d73c9e6-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:34:58.639103 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.639065    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/946311cc-f655-4798-bce5-10ad7d73c9e6-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:34:58.640326 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.640299    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/946311cc-f655-4798-bce5-10ad7d73c9e6-istio-podinfo" pod "946311cc-f655-4798-bce5-10ad7d73c9e6" (UID: "946311cc-f655-4798-bce5-10ad7d73c9e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:34:58.640458 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.640332    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/946311cc-f655-4798-bce5-10ad7d73c9e6-kube-api-access-n4sdd" pod "946311cc-f655-4798-bce5-10ad7d73c9e6" (UID: "946311cc-f655-4798-bce5-10ad7d73c9e6"). InnerVolumeSpecName "kube-api-access-n4sdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:34:58.640599 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.640578    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/946311cc-f655-4798-bce5-10ad7d73c9e6-istio-token" pod "946311cc-f655-4798-bce5-10ad7d73c9e6" (UID: "946311cc-f655-4798-bce5-10ad7d73c9e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:34:58.640599 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.640588    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/946311cc-f655-4798-bce5-10ad7d73c9e6-istio-envoy" pod "946311cc-f655-4798-bce5-10ad7d73c9e6" (UID: "946311cc-f655-4798-bce5-10ad7d73c9e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:34:58.680570 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.680541    4433 generic.go:356] "Generic (PLEG): container finished" podID="946311cc-f655-4798-bce5-10ad7d73c9e6" containerID="f2cbcd6bafafd3e966f436bf1923a9997029bcb4c93a7058cc01ee3e463ab3b4" exitCode=0
Oct 30 17:34:58.680721 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.680590    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"
Oct 30 17:34:58.680721 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.680624    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp" event={"ID":"946311cc-f655-4798-bce5-10ad7d73c9e6","Type":"ContainerDied","Data":"f2cbcd6bafafd3e966f436bf1923a9997029bcb4c93a7058cc01ee3e463ab3b4"}
Oct 30 17:34:58.680721 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.680687    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp" event={"ID":"946311cc-f655-4798-bce5-10ad7d73c9e6","Type":"ContainerDied","Data":"434fbe079508e89f7752fc54cf6937d29730bfd988f66d85382c9d19d83efab0"}
Oct 30 17:34:58.680721 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.680716    4433 scope.go:122] "RemoveContainer" containerID="f2cbcd6bafafd3e966f436bf1923a9997029bcb4c93a7058cc01ee3e463ab3b4"
Oct 30 17:34:58.691906 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.691886    4433 scope.go:122] "RemoveContainer" containerID="f2cbcd6bafafd3e966f436bf1923a9997029bcb4c93a7058cc01ee3e463ab3b4"
Oct 30 17:34:58.692196 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:34:58.692176    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2cbcd6bafafd3e966f436bf1923a9997029bcb4c93a7058cc01ee3e463ab3b4\": container with ID starting with f2cbcd6bafafd3e966f436bf1923a9997029bcb4c93a7058cc01ee3e463ab3b4 not found: ID does not exist" containerID="f2cbcd6bafafd3e966f436bf1923a9997029bcb4c93a7058cc01ee3e463ab3b4"
Oct 30 17:34:58.692257 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.692205    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2cbcd6bafafd3e966f436bf1923a9997029bcb4c93a7058cc01ee3e463ab3b4"} err="failed to get container status \"f2cbcd6bafafd3e966f436bf1923a9997029bcb4c93a7058cc01ee3e463ab3b4\": rpc error: code = NotFound desc = could not find container \"f2cbcd6bafafd3e966f436bf1923a9997029bcb4c93a7058cc01ee3e463ab3b4\": container with ID starting with f2cbcd6bafafd3e966f436bf1923a9997029bcb4c93a7058cc01ee3e463ab3b4 not found: ID does not exist"
Oct 30 17:34:58.733276 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.733252    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"]
Oct 30 17:34:58.737990 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.737937    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-hpvbp"]
Oct 30 17:34:58.740536 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.740515    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/946311cc-f655-4798-bce5-10ad7d73c9e6-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:34:58.740606 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.740538    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4sdd\" (UniqueName: \"kubernetes.io/projected/946311cc-f655-4798-bce5-10ad7d73c9e6-kube-api-access-n4sdd\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:34:58.740606 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.740548    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/946311cc-f655-4798-bce5-10ad7d73c9e6-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:34:58.740606 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:58.740558    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/946311cc-f655-4798-bce5-10ad7d73c9e6-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:34:59.695497 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:34:59.695465    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="946311cc-f655-4798-bce5-10ad7d73c9e6" path="/var/lib/kubelet/pods/946311cc-f655-4798-bce5-10ad7d73c9e6/volumes"
Oct 30 17:35:00.984582 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:00.984542    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v"]
Oct 30 17:35:00.985079 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:00.984799    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v" podUID="1b3d63c8-3b08-4283-a20a-7cfae44ae212" containerName="maas-api" containerID="cri-o://6b414db53ec388e198636ffcab1b3817bf05050aa87bf17b2cddc7b014b2d972" gracePeriod=30
Oct 30 17:35:01.306779 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:01.306745    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v"
Oct 30 17:35:01.395103 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:01.395056    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b3d63c8-3b08-4283-a20a-7cfae44ae212-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b3d63c8-3b08-4283-a20a-7cfae44ae212-openshift-service-ca\") pod \"1b3d63c8-3b08-4283-a20a-7cfae44ae212\" (UID: \"1b3d63c8-3b08-4283-a20a-7cfae44ae212\") "
Oct 30 17:35:01.395304 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:01.395192    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b3d63c8-3b08-4283-a20a-7cfae44ae212-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b3d63c8-3b08-4283-a20a-7cfae44ae212-maas-api-tls\") pod \"1b3d63c8-3b08-4283-a20a-7cfae44ae212\" (UID: \"1b3d63c8-3b08-4283-a20a-7cfae44ae212\") "
Oct 30 17:35:01.395304 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:01.395281    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b3d63c8-3b08-4283-a20a-7cfae44ae212-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b3d63c8-3b08-4283-a20a-7cfae44ae212-metrics-certs\") pod \"1b3d63c8-3b08-4283-a20a-7cfae44ae212\" (UID: \"1b3d63c8-3b08-4283-a20a-7cfae44ae212\") "
Oct 30 17:35:01.395490 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:01.395361    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b3d63c8-3b08-4283-a20a-7cfae44ae212-kube-api-access-zvwdr\" (UniqueName: \"kubernetes.io/projected/1b3d63c8-3b08-4283-a20a-7cfae44ae212-kube-api-access-zvwdr\") pod \"1b3d63c8-3b08-4283-a20a-7cfae44ae212\" (UID: \"1b3d63c8-3b08-4283-a20a-7cfae44ae212\") "
Oct 30 17:35:01.396228 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:01.396177    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b3d63c8-3b08-4283-a20a-7cfae44ae212-openshift-service-ca" pod "1b3d63c8-3b08-4283-a20a-7cfae44ae212" (UID: "1b3d63c8-3b08-4283-a20a-7cfae44ae212"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:35:01.398753 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:01.398724    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b3d63c8-3b08-4283-a20a-7cfae44ae212-kube-api-access-zvwdr" pod "1b3d63c8-3b08-4283-a20a-7cfae44ae212" (UID: "1b3d63c8-3b08-4283-a20a-7cfae44ae212"). InnerVolumeSpecName "kube-api-access-zvwdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:35:01.399297 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:01.399274    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b3d63c8-3b08-4283-a20a-7cfae44ae212-metrics-certs" pod "1b3d63c8-3b08-4283-a20a-7cfae44ae212" (UID: "1b3d63c8-3b08-4283-a20a-7cfae44ae212"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:35:01.399909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:01.399873    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b3d63c8-3b08-4283-a20a-7cfae44ae212-maas-api-tls" pod "1b3d63c8-3b08-4283-a20a-7cfae44ae212" (UID: "1b3d63c8-3b08-4283-a20a-7cfae44ae212"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:35:01.511692 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:01.506761    4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b3d63c8-3b08-4283-a20a-7cfae44ae212-openshift-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:01.511692 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:01.506803    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b3d63c8-3b08-4283-a20a-7cfae44ae212-maas-api-tls\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:01.511692 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:01.506817    4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b3d63c8-3b08-4283-a20a-7cfae44ae212-metrics-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:01.511692 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:01.506833    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvwdr\" (UniqueName: \"kubernetes.io/projected/1b3d63c8-3b08-4283-a20a-7cfae44ae212-kube-api-access-zvwdr\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:01.856281 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:01.856229    4433 generic.go:356] "Generic (PLEG): container finished" podID="1b3d63c8-3b08-4283-a20a-7cfae44ae212" containerID="6b414db53ec388e198636ffcab1b3817bf05050aa87bf17b2cddc7b014b2d972" exitCode=0
Oct 30 17:35:01.856830 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:01.856454    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v" event={"ID":"1b3d63c8-3b08-4283-a20a-7cfae44ae212","Type":"ContainerDied","Data":"6b414db53ec388e198636ffcab1b3817bf05050aa87bf17b2cddc7b014b2d972"}
Oct 30 17:35:01.857067 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:01.857035    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v" event={"ID":"1b3d63c8-3b08-4283-a20a-7cfae44ae212","Type":"ContainerDied","Data":"1503558f99d67000aec0be606459ea46913f0a781a35711fd402ee96b0b793cb"}
Oct 30 17:35:01.857234 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:01.856489    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v"
Oct 30 17:35:01.857861 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:01.857836    4433 scope.go:122] "RemoveContainer" containerID="6b414db53ec388e198636ffcab1b3817bf05050aa87bf17b2cddc7b014b2d972"
Oct 30 17:35:01.905676 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:01.905567    4433 scope.go:122] "RemoveContainer" containerID="6b414db53ec388e198636ffcab1b3817bf05050aa87bf17b2cddc7b014b2d972"
Oct 30 17:35:01.906119 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:35:01.906093    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b414db53ec388e198636ffcab1b3817bf05050aa87bf17b2cddc7b014b2d972\": container with ID starting with 6b414db53ec388e198636ffcab1b3817bf05050aa87bf17b2cddc7b014b2d972 not found: ID does not exist" containerID="6b414db53ec388e198636ffcab1b3817bf05050aa87bf17b2cddc7b014b2d972"
Oct 30 17:35:01.906188 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:01.906147    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b414db53ec388e198636ffcab1b3817bf05050aa87bf17b2cddc7b014b2d972"} err="failed to get container status \"6b414db53ec388e198636ffcab1b3817bf05050aa87bf17b2cddc7b014b2d972\": rpc error: code = NotFound desc = could not find container \"6b414db53ec388e198636ffcab1b3817bf05050aa87bf17b2cddc7b014b2d972\": container with ID starting with 6b414db53ec388e198636ffcab1b3817bf05050aa87bf17b2cddc7b014b2d972 not found: ID does not exist"
Oct 30 17:35:01.927484 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:01.927450    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v"]
Oct 30 17:35:01.937502 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:01.937472    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5d8947-7c5d98c5c9-4j65v"]
Oct 30 17:35:03.696737 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:03.696701    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b3d63c8-3b08-4283-a20a-7cfae44ae212" path="/var/lib/kubelet/pods/1b3d63c8-3b08-4283-a20a-7cfae44ae212/volumes"
Oct 30 17:35:25.591210 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:25.591166    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"]
Oct 30 17:35:25.591950 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:25.591452    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh" podUID="d99bf885-403c-47ec-9b74-9c4c57d0e3ef" containerName="istio-proxy" containerID="cri-o://c1f033efbd7946641a48ce9aa2746a441c5e2a4acfcf9142135a7aabd8959a02" gracePeriod=30
Oct 30 17:35:25.603055 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:25.603019    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"]
Oct 30 17:35:25.603312 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:25.603255    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc" podUID="367df923-d896-440c-8644-86fbc2f6edaf" containerName="istio-proxy" containerID="cri-o://22582ef841c0ce5175a70e6194da83c92429580924eb483c6c05119dee717273" gracePeriod=30
Oct 30 17:35:25.607891 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:25.607861    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"]
Oct 30 17:35:25.608140 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:25.608107    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894" podUID="5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9" containerName="istio-proxy" containerID="cri-o://bf8113882c8ca5815be335259b2723f5bc4498ad0c1469616ebfe5b1e7c91ab1" gracePeriod=30
Oct 30 17:35:26.979504 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:35:26.979461    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:35:28.328808 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:28.328767    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"]
Oct 30 17:35:28.329339 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:28.329050    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg" podUID="34119e6e-443a-40df-a54a-0f03b4e34553" containerName="istio-proxy" containerID="cri-o://d42a603f0a2c6d073f6d62a247feb87294dc3ddd4933800e8df490589a1d8c1b" gracePeriod=30
Oct 30 17:35:30.950100 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:30.950067    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"
Oct 30 17:35:30.953167 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:30.953141    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"
Oct 30 17:35:30.988064 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:30.988032    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"
Oct 30 17:35:31.034574 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.034526    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-istio-data\") pod \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") "
Oct 30 17:35:31.034853 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.034582    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/367df923-d896-440c-8644-86fbc2f6edaf-istio-token\" (UniqueName: \"kubernetes.io/projected/367df923-d896-440c-8644-86fbc2f6edaf-istio-token\") pod \"367df923-d896-440c-8644-86fbc2f6edaf\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") "
Oct 30 17:35:31.034853 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.034641    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-istio-podinfo\") pod \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") "
Oct 30 17:35:31.034853 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.034696    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-credential-socket\") pod \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") "
Oct 30 17:35:31.034853 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.034735    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-istio-data\") pod \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") "
Oct 30 17:35:31.034853 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.034763    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/367df923-d896-440c-8644-86fbc2f6edaf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/367df923-d896-440c-8644-86fbc2f6edaf-istio-podinfo\") pod \"367df923-d896-440c-8644-86fbc2f6edaf\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") "
Oct 30 17:35:31.034853 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.034851    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-workload-certs\") pod \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") "
Oct 30 17:35:31.035172 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.034863    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-istio-data" pod "5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9" (UID: "5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:35:31.035172 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.034887    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-workload-socket\") pod \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") "
Oct 30 17:35:31.035172 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.034920    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/367df923-d896-440c-8644-86fbc2f6edaf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/367df923-d896-440c-8644-86fbc2f6edaf-workload-socket\") pod \"367df923-d896-440c-8644-86fbc2f6edaf\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") "
Oct 30 17:35:31.035172 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.034977    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-istio-token\" (UniqueName: \"kubernetes.io/projected/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-istio-token\") pod \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") "
Oct 30 17:35:31.035172 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.035012    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/367df923-d896-440c-8644-86fbc2f6edaf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/367df923-d896-440c-8644-86fbc2f6edaf-istiod-ca-cert\") pod \"367df923-d896-440c-8644-86fbc2f6edaf\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") "
Oct 30 17:35:31.035172 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.035083    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-istio-data" pod "d99bf885-403c-47ec-9b74-9c4c57d0e3ef" (UID: "d99bf885-403c-47ec-9b74-9c4c57d0e3ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:35:31.035172 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.035086    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-credential-socket\") pod \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") "
Oct 30 17:35:31.035172 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.035133    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-istio-token\" (UniqueName: \"kubernetes.io/projected/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-istio-token\") pod \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") "
Oct 30 17:35:31.035172 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.035164    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-kube-api-access-dsn78\" (UniqueName: \"kubernetes.io/projected/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-kube-api-access-dsn78\") pod \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") "
Oct 30 17:35:31.035719 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.035195    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/367df923-d896-440c-8644-86fbc2f6edaf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/367df923-d896-440c-8644-86fbc2f6edaf-istio-data\") pod \"367df923-d896-440c-8644-86fbc2f6edaf\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") "
Oct 30 17:35:31.035719 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.035298    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-workload-socket" pod "d99bf885-403c-47ec-9b74-9c4c57d0e3ef" (UID: "d99bf885-403c-47ec-9b74-9c4c57d0e3ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:35:31.035719 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.035311    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-workload-socket\") pod \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") "
Oct 30 17:35:31.035719 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.035319    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-credential-socket" pod "5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9" (UID: "5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:35:31.035719 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.035357    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-kube-api-access-s84fp\" (UniqueName: \"kubernetes.io/projected/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-kube-api-access-s84fp\") pod \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") "
Oct 30 17:35:31.035719 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.035467    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-istiod-ca-cert\") pod \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") "
Oct 30 17:35:31.035719 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.035501    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/367df923-d896-440c-8644-86fbc2f6edaf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/367df923-d896-440c-8644-86fbc2f6edaf-istio-envoy\") pod \"367df923-d896-440c-8644-86fbc2f6edaf\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") "
Oct 30 17:35:31.035719 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.035530    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-istio-envoy\") pod \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") "
Oct 30 17:35:31.035719 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.035567    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-istiod-ca-cert\") pod \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") "
Oct 30 17:35:31.035719 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.035612    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-istio-envoy\") pod \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\" (UID: \"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9\") "
Oct 30 17:35:31.035719 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.035644    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-workload-certs\") pod \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") "
Oct 30 17:35:31.035719 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.035698    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/367df923-d896-440c-8644-86fbc2f6edaf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/367df923-d896-440c-8644-86fbc2f6edaf-workload-certs\") pod \"367df923-d896-440c-8644-86fbc2f6edaf\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") "
Oct 30 17:35:31.036387 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.035722    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367df923-d896-440c-8644-86fbc2f6edaf-istio-data" pod "367df923-d896-440c-8644-86fbc2f6edaf" (UID: "367df923-d896-440c-8644-86fbc2f6edaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:35:31.036387 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.035745    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/367df923-d896-440c-8644-86fbc2f6edaf-kube-api-access-m7bb8\" (UniqueName: \"kubernetes.io/projected/367df923-d896-440c-8644-86fbc2f6edaf-kube-api-access-m7bb8\") pod \"367df923-d896-440c-8644-86fbc2f6edaf\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") "
Oct 30 17:35:31.036387 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.035750    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/367df923-d896-440c-8644-86fbc2f6edaf-istiod-ca-cert" pod "367df923-d896-440c-8644-86fbc2f6edaf" (UID: "367df923-d896-440c-8644-86fbc2f6edaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:35:31.036387 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.035781    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-istio-podinfo\") pod \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\" (UID: \"d99bf885-403c-47ec-9b74-9c4c57d0e3ef\") "
Oct 30 17:35:31.036387 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.035892    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/367df923-d896-440c-8644-86fbc2f6edaf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/367df923-d896-440c-8644-86fbc2f6edaf-credential-socket\") pod \"367df923-d896-440c-8644-86fbc2f6edaf\" (UID: \"367df923-d896-440c-8644-86fbc2f6edaf\") "
Oct 30 17:35:31.036387 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.035963    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367df923-d896-440c-8644-86fbc2f6edaf-workload-socket" pod "367df923-d896-440c-8644-86fbc2f6edaf" (UID: "367df923-d896-440c-8644-86fbc2f6edaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:35:31.036387 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.035970    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-credential-socket" pod "d99bf885-403c-47ec-9b74-9c4c57d0e3ef" (UID: "d99bf885-403c-47ec-9b74-9c4c57d0e3ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:35:31.036387 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.036339    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-istiod-ca-cert" pod "5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9" (UID: "5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:35:31.036831 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.036578    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367df923-d896-440c-8644-86fbc2f6edaf-credential-socket" pod "367df923-d896-440c-8644-86fbc2f6edaf" (UID: "367df923-d896-440c-8644-86fbc2f6edaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:35:31.036831 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.036702    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-istiod-ca-cert" pod "d99bf885-403c-47ec-9b74-9c4c57d0e3ef" (UID: "d99bf885-403c-47ec-9b74-9c4c57d0e3ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:35:31.036831 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.036780    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/367df923-d896-440c-8644-86fbc2f6edaf-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:31.036831 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.036799    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:31.036831 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.036813    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:31.036831 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.036826    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:31.037171 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.036869    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:31.037171 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.036884    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/367df923-d896-440c-8644-86fbc2f6edaf-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:31.037171 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.036899    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/367df923-d896-440c-8644-86fbc2f6edaf-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:31.037171 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.036913    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:31.037171 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.036927    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/367df923-d896-440c-8644-86fbc2f6edaf-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:31.037171 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.036940    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:31.037171 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.036972    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-workload-socket" pod "5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9" (UID: "5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:35:31.038100 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.037697    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-workload-certs" pod "d99bf885-403c-47ec-9b74-9c4c57d0e3ef" (UID: "d99bf885-403c-47ec-9b74-9c4c57d0e3ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:35:31.038100 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.037875    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367df923-d896-440c-8644-86fbc2f6edaf-workload-certs" pod "367df923-d896-440c-8644-86fbc2f6edaf" (UID: "367df923-d896-440c-8644-86fbc2f6edaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:35:31.038817 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.038712    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-workload-certs" pod "5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9" (UID: "5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:35:31.038817 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.038789    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-istio-podinfo" pod "5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9" (UID: "5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:35:31.039468 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.039442    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/367df923-d896-440c-8644-86fbc2f6edaf-istio-token" pod "367df923-d896-440c-8644-86fbc2f6edaf" (UID: "367df923-d896-440c-8644-86fbc2f6edaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:35:31.040703 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.040667    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367df923-d896-440c-8644-86fbc2f6edaf-istio-envoy" pod "367df923-d896-440c-8644-86fbc2f6edaf" (UID: "367df923-d896-440c-8644-86fbc2f6edaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:35:31.042244 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.042199    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-istio-envoy" pod "d99bf885-403c-47ec-9b74-9c4c57d0e3ef" (UID: "d99bf885-403c-47ec-9b74-9c4c57d0e3ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:35:31.042244 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.042221    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/367df923-d896-440c-8644-86fbc2f6edaf-kube-api-access-m7bb8" pod "367df923-d896-440c-8644-86fbc2f6edaf" (UID: "367df923-d896-440c-8644-86fbc2f6edaf"). InnerVolumeSpecName "kube-api-access-m7bb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:35:31.042244 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.042203    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-kube-api-access-s84fp" pod "5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9" (UID: "5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9"). InnerVolumeSpecName "kube-api-access-s84fp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:35:31.042788 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.042366    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-kube-api-access-dsn78" pod "d99bf885-403c-47ec-9b74-9c4c57d0e3ef" (UID: "d99bf885-403c-47ec-9b74-9c4c57d0e3ef"). InnerVolumeSpecName "kube-api-access-dsn78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:35:31.042788 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.042522    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-istio-envoy" pod "5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9" (UID: "5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:35:31.043206 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.043169    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-istio-token" pod "d99bf885-403c-47ec-9b74-9c4c57d0e3ef" (UID: "d99bf885-403c-47ec-9b74-9c4c57d0e3ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:35:31.043486 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.043451    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/367df923-d896-440c-8644-86fbc2f6edaf-istio-podinfo" pod "367df923-d896-440c-8644-86fbc2f6edaf" (UID: "367df923-d896-440c-8644-86fbc2f6edaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:35:31.043813 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.043788    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-istio-token" pod "5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9" (UID: "5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:35:31.044036 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.044015    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-istio-podinfo" pod "d99bf885-403c-47ec-9b74-9c4c57d0e3ef" (UID: "d99bf885-403c-47ec-9b74-9c4c57d0e3ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:35:31.138556 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.138452    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:31.138556 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.138487    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsn78\" (UniqueName: \"kubernetes.io/projected/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-kube-api-access-dsn78\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:31.138556 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.138499    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:31.138556 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.138508    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s84fp\" (UniqueName: \"kubernetes.io/projected/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-kube-api-access-s84fp\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:31.138556 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.138517    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:31.138556 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.138526    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/367df923-d896-440c-8644-86fbc2f6edaf-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:31.138556 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.138533    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:31.138556 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.138541    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:31.138556 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.138549    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:31.138556 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.138557    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/367df923-d896-440c-8644-86fbc2f6edaf-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:31.138556 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.138567    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7bb8\" (UniqueName: \"kubernetes.io/projected/367df923-d896-440c-8644-86fbc2f6edaf-kube-api-access-m7bb8\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:31.139279 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.138576    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:31.139279 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.138585    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/367df923-d896-440c-8644-86fbc2f6edaf-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:31.139279 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.138592    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:31.139279 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.138599    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/367df923-d896-440c-8644-86fbc2f6edaf-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:31.139279 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.138606    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:31.139279 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.138613    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d99bf885-403c-47ec-9b74-9c4c57d0e3ef-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:31.169511 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.169476    4433 generic.go:356] "Generic (PLEG): container finished" podID="5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9" containerID="bf8113882c8ca5815be335259b2723f5bc4498ad0c1469616ebfe5b1e7c91ab1" exitCode=0
Oct 30 17:35:31.169715 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.169563    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"
Oct 30 17:35:31.169715 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.169603    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894" event={"ID":"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9","Type":"ContainerDied","Data":"bf8113882c8ca5815be335259b2723f5bc4498ad0c1469616ebfe5b1e7c91ab1"}
Oct 30 17:35:31.169715 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.169673    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894" event={"ID":"5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9","Type":"ContainerDied","Data":"2d7cc86de53c506abc2cf0c7ffb30de6b4150f1062fac961b504f792e76ddf71"}
Oct 30 17:35:31.169715 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.169696    4433 scope.go:122] "RemoveContainer" containerID="bf8113882c8ca5815be335259b2723f5bc4498ad0c1469616ebfe5b1e7c91ab1"
Oct 30 17:35:31.171203 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.171174    4433 generic.go:356] "Generic (PLEG): container finished" podID="d99bf885-403c-47ec-9b74-9c4c57d0e3ef" containerID="c1f033efbd7946641a48ce9aa2746a441c5e2a4acfcf9142135a7aabd8959a02" exitCode=0
Oct 30 17:35:31.171339 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.171206    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh" event={"ID":"d99bf885-403c-47ec-9b74-9c4c57d0e3ef","Type":"ContainerDied","Data":"c1f033efbd7946641a48ce9aa2746a441c5e2a4acfcf9142135a7aabd8959a02"}
Oct 30 17:35:31.171339 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.171221    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"
Oct 30 17:35:31.171339 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.171261    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh" event={"ID":"d99bf885-403c-47ec-9b74-9c4c57d0e3ef","Type":"ContainerDied","Data":"bd4a48e4088870d7dd6f3d3f9cbe747b70d0c7b1f97de81aaa52360fa7034dc9"}
Oct 30 17:35:31.172894 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.172866    4433 generic.go:356] "Generic (PLEG): container finished" podID="367df923-d896-440c-8644-86fbc2f6edaf" containerID="22582ef841c0ce5175a70e6194da83c92429580924eb483c6c05119dee717273" exitCode=0
Oct 30 17:35:31.173029 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.172914    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"
Oct 30 17:35:31.173029 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.172947    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc" event={"ID":"367df923-d896-440c-8644-86fbc2f6edaf","Type":"ContainerDied","Data":"22582ef841c0ce5175a70e6194da83c92429580924eb483c6c05119dee717273"}
Oct 30 17:35:31.173029 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.172999    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc" event={"ID":"367df923-d896-440c-8644-86fbc2f6edaf","Type":"ContainerDied","Data":"c967f0daf90ddf6a1e9b8ad5648526ae812b543b14215e2bc46dbcad46ed1f7e"}
Oct 30 17:35:31.182626 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.182377    4433 scope.go:122] "RemoveContainer" containerID="bf8113882c8ca5815be335259b2723f5bc4498ad0c1469616ebfe5b1e7c91ab1"
Oct 30 17:35:31.182793 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:35:31.182734    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf8113882c8ca5815be335259b2723f5bc4498ad0c1469616ebfe5b1e7c91ab1\": container with ID starting with bf8113882c8ca5815be335259b2723f5bc4498ad0c1469616ebfe5b1e7c91ab1 not found: ID does not exist" containerID="bf8113882c8ca5815be335259b2723f5bc4498ad0c1469616ebfe5b1e7c91ab1"
Oct 30 17:35:31.182793 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.182768    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf8113882c8ca5815be335259b2723f5bc4498ad0c1469616ebfe5b1e7c91ab1"} err="failed to get container status \"bf8113882c8ca5815be335259b2723f5bc4498ad0c1469616ebfe5b1e7c91ab1\": rpc error: code = NotFound desc = could not find container \"bf8113882c8ca5815be335259b2723f5bc4498ad0c1469616ebfe5b1e7c91ab1\": container with ID starting with bf8113882c8ca5815be335259b2723f5bc4498ad0c1469616ebfe5b1e7c91ab1 not found: ID does not exist"
Oct 30 17:35:31.182923 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.182792    4433 scope.go:122] "RemoveContainer" containerID="c1f033efbd7946641a48ce9aa2746a441c5e2a4acfcf9142135a7aabd8959a02"
Oct 30 17:35:31.193631 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.193608    4433 scope.go:122] "RemoveContainer" containerID="c1f033efbd7946641a48ce9aa2746a441c5e2a4acfcf9142135a7aabd8959a02"
Oct 30 17:35:31.193956 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:35:31.193931    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1f033efbd7946641a48ce9aa2746a441c5e2a4acfcf9142135a7aabd8959a02\": container with ID starting with c1f033efbd7946641a48ce9aa2746a441c5e2a4acfcf9142135a7aabd8959a02 not found: ID does not exist" containerID="c1f033efbd7946641a48ce9aa2746a441c5e2a4acfcf9142135a7aabd8959a02"
Oct 30 17:35:31.194042 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.193969    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1f033efbd7946641a48ce9aa2746a441c5e2a4acfcf9142135a7aabd8959a02"} err="failed to get container status \"c1f033efbd7946641a48ce9aa2746a441c5e2a4acfcf9142135a7aabd8959a02\": rpc error: code = NotFound desc = could not find container \"c1f033efbd7946641a48ce9aa2746a441c5e2a4acfcf9142135a7aabd8959a02\": container with ID starting with c1f033efbd7946641a48ce9aa2746a441c5e2a4acfcf9142135a7aabd8959a02 not found: ID does not exist"
Oct 30 17:35:31.194042 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.193994    4433 scope.go:122] "RemoveContainer" containerID="22582ef841c0ce5175a70e6194da83c92429580924eb483c6c05119dee717273"
Oct 30 17:35:31.204465 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.204444    4433 scope.go:122] "RemoveContainer" containerID="22582ef841c0ce5175a70e6194da83c92429580924eb483c6c05119dee717273"
Oct 30 17:35:31.204800 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:35:31.204779    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22582ef841c0ce5175a70e6194da83c92429580924eb483c6c05119dee717273\": container with ID starting with 22582ef841c0ce5175a70e6194da83c92429580924eb483c6c05119dee717273 not found: ID does not exist" containerID="22582ef841c0ce5175a70e6194da83c92429580924eb483c6c05119dee717273"
Oct 30 17:35:31.204871 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.204822    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22582ef841c0ce5175a70e6194da83c92429580924eb483c6c05119dee717273"} err="failed to get container status \"22582ef841c0ce5175a70e6194da83c92429580924eb483c6c05119dee717273\": rpc error: code = NotFound desc = could not find container \"22582ef841c0ce5175a70e6194da83c92429580924eb483c6c05119dee717273\": container with ID starting with 22582ef841c0ce5175a70e6194da83c92429580924eb483c6c05119dee717273 not found: ID does not exist"
Oct 30 17:35:31.237037 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.237001    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"]
Oct 30 17:35:31.242035 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.242006    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-cb894"]
Oct 30 17:35:31.289602 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.289569    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"]
Oct 30 17:35:31.294521 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.294487    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-j45dh"]
Oct 30 17:35:31.328658 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.328613    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"]
Oct 30 17:35:31.331667 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.331644    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-464tc"]
Oct 30 17:35:31.695873 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.695829    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="367df923-d896-440c-8644-86fbc2f6edaf" path="/var/lib/kubelet/pods/367df923-d896-440c-8644-86fbc2f6edaf/volumes"
Oct 30 17:35:31.696521 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.696499    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9" path="/var/lib/kubelet/pods/5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9/volumes"
Oct 30 17:35:31.696992 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:31.696978    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d99bf885-403c-47ec-9b74-9c4c57d0e3ef" path="/var/lib/kubelet/pods/d99bf885-403c-47ec-9b74-9c4c57d0e3ef/volumes"
Oct 30 17:35:33.616298 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:33.616273    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:35:33.797494 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:33.797456    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34119e6e-443a-40df-a54a-0f03b4e34553-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34119e6e-443a-40df-a54a-0f03b4e34553-workload-socket\") pod \"34119e6e-443a-40df-a54a-0f03b4e34553\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") "
Oct 30 17:35:33.797683 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:33.797507    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34119e6e-443a-40df-a54a-0f03b4e34553-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34119e6e-443a-40df-a54a-0f03b4e34553-istio-podinfo\") pod \"34119e6e-443a-40df-a54a-0f03b4e34553\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") "
Oct 30 17:35:33.797683 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:33.797556    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34119e6e-443a-40df-a54a-0f03b4e34553-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34119e6e-443a-40df-a54a-0f03b4e34553-istiod-ca-cert\") pod \"34119e6e-443a-40df-a54a-0f03b4e34553\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") "
Oct 30 17:35:33.797683 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:33.797583    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34119e6e-443a-40df-a54a-0f03b4e34553-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34119e6e-443a-40df-a54a-0f03b4e34553-credential-socket\") pod \"34119e6e-443a-40df-a54a-0f03b4e34553\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") "
Oct 30 17:35:33.797683 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:33.797641    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34119e6e-443a-40df-a54a-0f03b4e34553-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34119e6e-443a-40df-a54a-0f03b4e34553-istio-data\") pod \"34119e6e-443a-40df-a54a-0f03b4e34553\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") "
Oct 30 17:35:33.797683 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:33.797670    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34119e6e-443a-40df-a54a-0f03b4e34553-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34119e6e-443a-40df-a54a-0f03b4e34553-istio-envoy\") pod \"34119e6e-443a-40df-a54a-0f03b4e34553\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") "
Oct 30 17:35:33.797967 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:33.797709    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34119e6e-443a-40df-a54a-0f03b4e34553-kube-api-access-8t7kz\" (UniqueName: \"kubernetes.io/projected/34119e6e-443a-40df-a54a-0f03b4e34553-kube-api-access-8t7kz\") pod \"34119e6e-443a-40df-a54a-0f03b4e34553\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") "
Oct 30 17:35:33.797967 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:33.797787    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34119e6e-443a-40df-a54a-0f03b4e34553-workload-socket" pod "34119e6e-443a-40df-a54a-0f03b4e34553" (UID: "34119e6e-443a-40df-a54a-0f03b4e34553"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:35:33.797967 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:33.797864    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34119e6e-443a-40df-a54a-0f03b4e34553-istio-token\" (UniqueName: \"kubernetes.io/projected/34119e6e-443a-40df-a54a-0f03b4e34553-istio-token\") pod \"34119e6e-443a-40df-a54a-0f03b4e34553\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") "
Oct 30 17:35:33.798134 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:33.797989    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34119e6e-443a-40df-a54a-0f03b4e34553-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34119e6e-443a-40df-a54a-0f03b4e34553-workload-certs\") pod \"34119e6e-443a-40df-a54a-0f03b4e34553\" (UID: \"34119e6e-443a-40df-a54a-0f03b4e34553\") "
Oct 30 17:35:33.798134 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:33.798020    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34119e6e-443a-40df-a54a-0f03b4e34553-credential-socket" pod "34119e6e-443a-40df-a54a-0f03b4e34553" (UID: "34119e6e-443a-40df-a54a-0f03b4e34553"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:35:33.798134 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:33.798107    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34119e6e-443a-40df-a54a-0f03b4e34553-istio-data" pod "34119e6e-443a-40df-a54a-0f03b4e34553" (UID: "34119e6e-443a-40df-a54a-0f03b4e34553"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:35:33.798134 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:33.798114    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34119e6e-443a-40df-a54a-0f03b4e34553-istiod-ca-cert" pod "34119e6e-443a-40df-a54a-0f03b4e34553" (UID: "34119e6e-443a-40df-a54a-0f03b4e34553"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:35:33.798504 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:33.798477    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34119e6e-443a-40df-a54a-0f03b4e34553-workload-certs" pod "34119e6e-443a-40df-a54a-0f03b4e34553" (UID: "34119e6e-443a-40df-a54a-0f03b4e34553"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:35:33.798777 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:33.798743    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34119e6e-443a-40df-a54a-0f03b4e34553-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:33.798880 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:33.798786    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34119e6e-443a-40df-a54a-0f03b4e34553-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:33.798985 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:33.798934    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34119e6e-443a-40df-a54a-0f03b4e34553-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:33.799075 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:33.799052    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34119e6e-443a-40df-a54a-0f03b4e34553-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:33.799128 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:33.799085    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34119e6e-443a-40df-a54a-0f03b4e34553-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:33.800146 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:33.800114    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34119e6e-443a-40df-a54a-0f03b4e34553-istio-podinfo" pod "34119e6e-443a-40df-a54a-0f03b4e34553" (UID: "34119e6e-443a-40df-a54a-0f03b4e34553"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:35:33.800290 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:33.800270    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34119e6e-443a-40df-a54a-0f03b4e34553-istio-envoy" pod "34119e6e-443a-40df-a54a-0f03b4e34553" (UID: "34119e6e-443a-40df-a54a-0f03b4e34553"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:35:33.800346 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:33.800334    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34119e6e-443a-40df-a54a-0f03b4e34553-istio-token" pod "34119e6e-443a-40df-a54a-0f03b4e34553" (UID: "34119e6e-443a-40df-a54a-0f03b4e34553"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:35:33.800821 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:33.800805    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34119e6e-443a-40df-a54a-0f03b4e34553-kube-api-access-8t7kz" pod "34119e6e-443a-40df-a54a-0f03b4e34553" (UID: "34119e6e-443a-40df-a54a-0f03b4e34553"). InnerVolumeSpecName "kube-api-access-8t7kz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:35:33.900510 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:33.900424    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34119e6e-443a-40df-a54a-0f03b4e34553-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:33.900510 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:33.900458    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t7kz\" (UniqueName: \"kubernetes.io/projected/34119e6e-443a-40df-a54a-0f03b4e34553-kube-api-access-8t7kz\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:33.900510 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:33.900468    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34119e6e-443a-40df-a54a-0f03b4e34553-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:33.900510 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:33.900480    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34119e6e-443a-40df-a54a-0f03b4e34553-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:34.199978 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:34.199935    4433 generic.go:356] "Generic (PLEG): container finished" podID="34119e6e-443a-40df-a54a-0f03b4e34553" containerID="d42a603f0a2c6d073f6d62a247feb87294dc3ddd4933800e8df490589a1d8c1b" exitCode=0
Oct 30 17:35:34.200185 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:34.199997    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"
Oct 30 17:35:34.200185 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:34.200012    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg" event={"ID":"34119e6e-443a-40df-a54a-0f03b4e34553","Type":"ContainerDied","Data":"d42a603f0a2c6d073f6d62a247feb87294dc3ddd4933800e8df490589a1d8c1b"}
Oct 30 17:35:34.200185 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:34.200091    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg" event={"ID":"34119e6e-443a-40df-a54a-0f03b4e34553","Type":"ContainerDied","Data":"56db25acc3ea20484cd332afdacc5ccb855f1c7c3ee4a4ad52844ef9e52271f3"}
Oct 30 17:35:34.200185 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:34.200118    4433 scope.go:122] "RemoveContainer" containerID="d42a603f0a2c6d073f6d62a247feb87294dc3ddd4933800e8df490589a1d8c1b"
Oct 30 17:35:34.211438 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:34.211415    4433 scope.go:122] "RemoveContainer" containerID="db7028eeec58108cdd61659d39cc8be05b3e56e49b004f2268765a17af8eb4c2"
Oct 30 17:35:34.221996 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:34.221968    4433 scope.go:122] "RemoveContainer" containerID="d42a603f0a2c6d073f6d62a247feb87294dc3ddd4933800e8df490589a1d8c1b"
Oct 30 17:35:34.222331 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:35:34.222307    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d42a603f0a2c6d073f6d62a247feb87294dc3ddd4933800e8df490589a1d8c1b\": container with ID starting with d42a603f0a2c6d073f6d62a247feb87294dc3ddd4933800e8df490589a1d8c1b not found: ID does not exist" containerID="d42a603f0a2c6d073f6d62a247feb87294dc3ddd4933800e8df490589a1d8c1b"
Oct 30 17:35:34.222399 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:34.222342    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d42a603f0a2c6d073f6d62a247feb87294dc3ddd4933800e8df490589a1d8c1b"} err="failed to get container status \"d42a603f0a2c6d073f6d62a247feb87294dc3ddd4933800e8df490589a1d8c1b\": rpc error: code = NotFound desc = could not find container \"d42a603f0a2c6d073f6d62a247feb87294dc3ddd4933800e8df490589a1d8c1b\": container with ID starting with d42a603f0a2c6d073f6d62a247feb87294dc3ddd4933800e8df490589a1d8c1b not found: ID does not exist"
Oct 30 17:35:34.222399 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:34.222361    4433 scope.go:122] "RemoveContainer" containerID="db7028eeec58108cdd61659d39cc8be05b3e56e49b004f2268765a17af8eb4c2"
Oct 30 17:35:34.222739 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:35:34.222717    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db7028eeec58108cdd61659d39cc8be05b3e56e49b004f2268765a17af8eb4c2\": container with ID starting with db7028eeec58108cdd61659d39cc8be05b3e56e49b004f2268765a17af8eb4c2 not found: ID does not exist" containerID="db7028eeec58108cdd61659d39cc8be05b3e56e49b004f2268765a17af8eb4c2"
Oct 30 17:35:34.222833 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:34.222742    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db7028eeec58108cdd61659d39cc8be05b3e56e49b004f2268765a17af8eb4c2"} err="failed to get container status \"db7028eeec58108cdd61659d39cc8be05b3e56e49b004f2268765a17af8eb4c2\": rpc error: code = NotFound desc = could not find container \"db7028eeec58108cdd61659d39cc8be05b3e56e49b004f2268765a17af8eb4c2\": container with ID starting with db7028eeec58108cdd61659d39cc8be05b3e56e49b004f2268765a17af8eb4c2 not found: ID does not exist"
Oct 30 17:35:34.252373 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:34.252340    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"]
Oct 30 17:35:34.264577 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:34.264540    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-9e9ce7-openshift-default-d99497d45-rvchg"]
Oct 30 17:35:35.695262 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:35.695226    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34119e6e-443a-40df-a54a-0f03b4e34553" path="/var/lib/kubelet/pods/34119e6e-443a-40df-a54a-0f03b4e34553/volumes"
Oct 30 17:35:36.235149 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:36.235114    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d4f99-8ff54d7d7-qp8mh"]
Oct 30 17:35:40.184427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:40.180813    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:35:41.235984 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:41.235906    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d4f99-8ff54d7d7-qp8mh" podUID="7741e951-b0c5-432a-ae92-8dc492b47cf7" containerName="payload-pre-processing" containerID="cri-o://0cc3129a188de66ea5439687f2cb29905f6ac21823ed69b5a1a325291a4060e9" gracePeriod=25
Oct 30 17:35:50.777949 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:50.777851    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d4f99-8ff54d7d7-qp8mh" podUID="7741e951-b0c5-432a-ae92-8dc492b47cf7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:35:53.700630 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:53.700595    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"]
Oct 30 17:35:53.701114 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:53.700856    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft" podUID="5fcae069-8f9f-47e5-8d3a-82c248cf3c31" containerName="istio-proxy" containerID="cri-o://887486d0abbe40860e456f4aafa560edc08e7ebb1b23608ee565746b02f71998" gracePeriod=30
Oct 30 17:35:55.648757 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:55.648713    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"]
Oct 30 17:35:55.649232 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:55.649027    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg" podUID="303728db-2515-4854-a0ca-e8726a6414f5" containerName="istio-proxy" containerID="cri-o://8f97e87ff243fa618bcea3b25297bd69d3ac3c710e362f714b1b5637249c17c3" gracePeriod=30
Oct 30 17:35:56.979736 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:35:56.979691    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:35:58.968443 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:58.968369    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:35:59.002357 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.002323    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-credential-socket\") pod \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") "
Oct 30 17:35:59.002562 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.002386    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-kube-api-access-qsn5w\" (UniqueName: \"kubernetes.io/projected/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-kube-api-access-qsn5w\") pod \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") "
Oct 30 17:35:59.002562 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.002485    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-workload-certs\") pod \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") "
Oct 30 17:35:59.002663 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.002580    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-workload-socket\") pod \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") "
Oct 30 17:35:59.002663 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.002616    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-istio-data\") pod \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") "
Oct 30 17:35:59.002781 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.002683    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-credential-socket" pod "5fcae069-8f9f-47e5-8d3a-82c248cf3c31" (UID: "5fcae069-8f9f-47e5-8d3a-82c248cf3c31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:35:59.002781 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.002692    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-istio-token\" (UniqueName: \"kubernetes.io/projected/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-istio-token\") pod \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") "
Oct 30 17:35:59.002898 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.002798    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-istio-podinfo\") pod \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") "
Oct 30 17:35:59.002898 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.002798    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-workload-certs" pod "5fcae069-8f9f-47e5-8d3a-82c248cf3c31" (UID: "5fcae069-8f9f-47e5-8d3a-82c248cf3c31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:35:59.002898 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.002856    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-istio-envoy\") pod \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") "
Oct 30 17:35:59.003070 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.002895    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-istiod-ca-cert\") pod \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\" (UID: \"5fcae069-8f9f-47e5-8d3a-82c248cf3c31\") "
Oct 30 17:35:59.003070 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.002880    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-workload-socket" pod "5fcae069-8f9f-47e5-8d3a-82c248cf3c31" (UID: "5fcae069-8f9f-47e5-8d3a-82c248cf3c31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:35:59.003070 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.002944    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-istio-data" pod "5fcae069-8f9f-47e5-8d3a-82c248cf3c31" (UID: "5fcae069-8f9f-47e5-8d3a-82c248cf3c31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:35:59.003387 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.003362    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:59.003555 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.003394    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:59.003555 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.003421    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:59.003555 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.003434    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:59.003555 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.003360    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-istiod-ca-cert" pod "5fcae069-8f9f-47e5-8d3a-82c248cf3c31" (UID: "5fcae069-8f9f-47e5-8d3a-82c248cf3c31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:35:59.004860 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.004831    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-kube-api-access-qsn5w" pod "5fcae069-8f9f-47e5-8d3a-82c248cf3c31" (UID: "5fcae069-8f9f-47e5-8d3a-82c248cf3c31"). InnerVolumeSpecName "kube-api-access-qsn5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:35:59.005061 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.005035    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-istio-envoy" pod "5fcae069-8f9f-47e5-8d3a-82c248cf3c31" (UID: "5fcae069-8f9f-47e5-8d3a-82c248cf3c31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:35:59.005327 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.005303    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-istio-podinfo" pod "5fcae069-8f9f-47e5-8d3a-82c248cf3c31" (UID: "5fcae069-8f9f-47e5-8d3a-82c248cf3c31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:35:59.005327 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.005314    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-istio-token" pod "5fcae069-8f9f-47e5-8d3a-82c248cf3c31" (UID: "5fcae069-8f9f-47e5-8d3a-82c248cf3c31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:35:59.105181 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.105139    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsn5w\" (UniqueName: \"kubernetes.io/projected/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-kube-api-access-qsn5w\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:59.105181 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.105173    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:59.105181 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.105182    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:59.105181 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.105190    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:59.105495 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.105198    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fcae069-8f9f-47e5-8d3a-82c248cf3c31-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:35:59.482262 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.482227    4433 generic.go:356] "Generic (PLEG): container finished" podID="5fcae069-8f9f-47e5-8d3a-82c248cf3c31" containerID="887486d0abbe40860e456f4aafa560edc08e7ebb1b23608ee565746b02f71998" exitCode=0
Oct 30 17:35:59.482466 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.482301    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"
Oct 30 17:35:59.482466 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.482312    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft" event={"ID":"5fcae069-8f9f-47e5-8d3a-82c248cf3c31","Type":"ContainerDied","Data":"887486d0abbe40860e456f4aafa560edc08e7ebb1b23608ee565746b02f71998"}
Oct 30 17:35:59.482466 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.482423    4433 scope.go:122] "RemoveContainer" containerID="887486d0abbe40860e456f4aafa560edc08e7ebb1b23608ee565746b02f71998"
Oct 30 17:35:59.482611 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.482417    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft" event={"ID":"5fcae069-8f9f-47e5-8d3a-82c248cf3c31","Type":"ContainerDied","Data":"bb5779c2eb0baa5c0cd010863d9dea828667f1bf9d5338cb4f13e8edee9e6533"}
Oct 30 17:35:59.493377 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.493356    4433 scope.go:122] "RemoveContainer" containerID="0468bdc637faecdad7c1817dd916908c2917732fc4d55b05b4779401cb8f5afc"
Oct 30 17:35:59.502984 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.502964    4433 scope.go:122] "RemoveContainer" containerID="887486d0abbe40860e456f4aafa560edc08e7ebb1b23608ee565746b02f71998"
Oct 30 17:35:59.503290 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:35:59.503269    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"887486d0abbe40860e456f4aafa560edc08e7ebb1b23608ee565746b02f71998\": container with ID starting with 887486d0abbe40860e456f4aafa560edc08e7ebb1b23608ee565746b02f71998 not found: ID does not exist" containerID="887486d0abbe40860e456f4aafa560edc08e7ebb1b23608ee565746b02f71998"
Oct 30 17:35:59.503350 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.503299    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"887486d0abbe40860e456f4aafa560edc08e7ebb1b23608ee565746b02f71998"} err="failed to get container status \"887486d0abbe40860e456f4aafa560edc08e7ebb1b23608ee565746b02f71998\": rpc error: code = NotFound desc = could not find container \"887486d0abbe40860e456f4aafa560edc08e7ebb1b23608ee565746b02f71998\": container with ID starting with 887486d0abbe40860e456f4aafa560edc08e7ebb1b23608ee565746b02f71998 not found: ID does not exist"
Oct 30 17:35:59.503350 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.503318    4433 scope.go:122] "RemoveContainer" containerID="0468bdc637faecdad7c1817dd916908c2917732fc4d55b05b4779401cb8f5afc"
Oct 30 17:35:59.503593 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:35:59.503577    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0468bdc637faecdad7c1817dd916908c2917732fc4d55b05b4779401cb8f5afc\": container with ID starting with 0468bdc637faecdad7c1817dd916908c2917732fc4d55b05b4779401cb8f5afc not found: ID does not exist" containerID="0468bdc637faecdad7c1817dd916908c2917732fc4d55b05b4779401cb8f5afc"
Oct 30 17:35:59.503636 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.503597    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0468bdc637faecdad7c1817dd916908c2917732fc4d55b05b4779401cb8f5afc"} err="failed to get container status \"0468bdc637faecdad7c1817dd916908c2917732fc4d55b05b4779401cb8f5afc\": rpc error: code = NotFound desc = could not find container \"0468bdc637faecdad7c1817dd916908c2917732fc4d55b05b4779401cb8f5afc\": container with ID starting with 0468bdc637faecdad7c1817dd916908c2917732fc4d55b05b4779401cb8f5afc not found: ID does not exist"
Oct 30 17:35:59.536955 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.536920    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"]
Oct 30 17:35:59.540083 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.540061    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-4fb821-openshift-default-75c9b45cc9-xgjft"]
Oct 30 17:35:59.695631 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:35:59.695595    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fcae069-8f9f-47e5-8d3a-82c248cf3c31" path="/var/lib/kubelet/pods/5fcae069-8f9f-47e5-8d3a-82c248cf3c31/volumes"
Oct 30 17:36:00.070109 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:00.070069    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:36:00.070519 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:00.070129    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg" podUID="303728db-2515-4854-a0ca-e8726a6414f5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:36:00.777525 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:00.777490    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d4f99-8ff54d7d7-qp8mh" podUID="7741e951-b0c5-432a-ae92-8dc492b47cf7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:36:00.916844 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:00.916777    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"
Oct 30 17:36:01.031589 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.031556    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/303728db-2515-4854-a0ca-e8726a6414f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/303728db-2515-4854-a0ca-e8726a6414f5-credential-socket\") pod \"303728db-2515-4854-a0ca-e8726a6414f5\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") "
Oct 30 17:36:01.031799 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.031650    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/303728db-2515-4854-a0ca-e8726a6414f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/303728db-2515-4854-a0ca-e8726a6414f5-workload-socket\") pod \"303728db-2515-4854-a0ca-e8726a6414f5\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") "
Oct 30 17:36:01.031799 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.031670    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/303728db-2515-4854-a0ca-e8726a6414f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/303728db-2515-4854-a0ca-e8726a6414f5-istio-envoy\") pod \"303728db-2515-4854-a0ca-e8726a6414f5\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") "
Oct 30 17:36:01.031799 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.031718    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/303728db-2515-4854-a0ca-e8726a6414f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/303728db-2515-4854-a0ca-e8726a6414f5-istio-data\") pod \"303728db-2515-4854-a0ca-e8726a6414f5\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") "
Oct 30 17:36:01.031799 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.031750    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/303728db-2515-4854-a0ca-e8726a6414f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/303728db-2515-4854-a0ca-e8726a6414f5-istio-podinfo\") pod \"303728db-2515-4854-a0ca-e8726a6414f5\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") "
Oct 30 17:36:01.031799 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.031786    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/303728db-2515-4854-a0ca-e8726a6414f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/303728db-2515-4854-a0ca-e8726a6414f5-workload-certs\") pod \"303728db-2515-4854-a0ca-e8726a6414f5\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") "
Oct 30 17:36:01.032115 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.031846    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/303728db-2515-4854-a0ca-e8726a6414f5-istio-token\" (UniqueName: \"kubernetes.io/projected/303728db-2515-4854-a0ca-e8726a6414f5-istio-token\") pod \"303728db-2515-4854-a0ca-e8726a6414f5\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") "
Oct 30 17:36:01.032115 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.031859    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303728db-2515-4854-a0ca-e8726a6414f5-credential-socket" pod "303728db-2515-4854-a0ca-e8726a6414f5" (UID: "303728db-2515-4854-a0ca-e8726a6414f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:01.032115 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.031883    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/303728db-2515-4854-a0ca-e8726a6414f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/303728db-2515-4854-a0ca-e8726a6414f5-istiod-ca-cert\") pod \"303728db-2515-4854-a0ca-e8726a6414f5\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") "
Oct 30 17:36:01.032115 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.031949    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/303728db-2515-4854-a0ca-e8726a6414f5-kube-api-access-cjzpz\" (UniqueName: \"kubernetes.io/projected/303728db-2515-4854-a0ca-e8726a6414f5-kube-api-access-cjzpz\") pod \"303728db-2515-4854-a0ca-e8726a6414f5\" (UID: \"303728db-2515-4854-a0ca-e8726a6414f5\") "
Oct 30 17:36:01.032115 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.032003    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303728db-2515-4854-a0ca-e8726a6414f5-workload-socket" pod "303728db-2515-4854-a0ca-e8726a6414f5" (UID: "303728db-2515-4854-a0ca-e8726a6414f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:01.032399 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.032113    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303728db-2515-4854-a0ca-e8726a6414f5-workload-certs" pod "303728db-2515-4854-a0ca-e8726a6414f5" (UID: "303728db-2515-4854-a0ca-e8726a6414f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:01.032399 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.032149    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303728db-2515-4854-a0ca-e8726a6414f5-istio-data" pod "303728db-2515-4854-a0ca-e8726a6414f5" (UID: "303728db-2515-4854-a0ca-e8726a6414f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:01.032399 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.032390    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/303728db-2515-4854-a0ca-e8726a6414f5-istiod-ca-cert" pod "303728db-2515-4854-a0ca-e8726a6414f5" (UID: "303728db-2515-4854-a0ca-e8726a6414f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:36:01.032644 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.032601    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/303728db-2515-4854-a0ca-e8726a6414f5-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:01.032644 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.032622    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/303728db-2515-4854-a0ca-e8726a6414f5-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:01.032644 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.032637    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/303728db-2515-4854-a0ca-e8726a6414f5-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:01.032781 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.032650    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/303728db-2515-4854-a0ca-e8726a6414f5-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:01.032781 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.032663    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/303728db-2515-4854-a0ca-e8726a6414f5-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:01.034165 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.034141    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/303728db-2515-4854-a0ca-e8726a6414f5-istio-token" pod "303728db-2515-4854-a0ca-e8726a6414f5" (UID: "303728db-2515-4854-a0ca-e8726a6414f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:36:01.034295 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.034275    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/303728db-2515-4854-a0ca-e8726a6414f5-kube-api-access-cjzpz" pod "303728db-2515-4854-a0ca-e8726a6414f5" (UID: "303728db-2515-4854-a0ca-e8726a6414f5"). InnerVolumeSpecName "kube-api-access-cjzpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:36:01.034449 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.034430    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303728db-2515-4854-a0ca-e8726a6414f5-istio-envoy" pod "303728db-2515-4854-a0ca-e8726a6414f5" (UID: "303728db-2515-4854-a0ca-e8726a6414f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:01.034501 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.034455    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/303728db-2515-4854-a0ca-e8726a6414f5-istio-podinfo" pod "303728db-2515-4854-a0ca-e8726a6414f5" (UID: "303728db-2515-4854-a0ca-e8726a6414f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:36:01.134476 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.134397    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/303728db-2515-4854-a0ca-e8726a6414f5-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:01.134476 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.134461    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/303728db-2515-4854-a0ca-e8726a6414f5-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:01.134476 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.134470    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/303728db-2515-4854-a0ca-e8726a6414f5-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:01.134476 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.134479    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjzpz\" (UniqueName: \"kubernetes.io/projected/303728db-2515-4854-a0ca-e8726a6414f5-kube-api-access-cjzpz\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:01.500340 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.500302    4433 generic.go:356] "Generic (PLEG): container finished" podID="303728db-2515-4854-a0ca-e8726a6414f5" containerID="8f97e87ff243fa618bcea3b25297bd69d3ac3c710e362f714b1b5637249c17c3" exitCode=0
Oct 30 17:36:01.500567 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.500358    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"
Oct 30 17:36:01.500567 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.500351    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg" event={"ID":"303728db-2515-4854-a0ca-e8726a6414f5","Type":"ContainerDied","Data":"8f97e87ff243fa618bcea3b25297bd69d3ac3c710e362f714b1b5637249c17c3"}
Oct 30 17:36:01.500567 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.500494    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg" event={"ID":"303728db-2515-4854-a0ca-e8726a6414f5","Type":"ContainerDied","Data":"3aa94546cce0276d07adfe37e9ba5d1c1f98a0999a5c6bfa76478f9fe9c4c220"}
Oct 30 17:36:01.500567 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.500534    4433 scope.go:122] "RemoveContainer" containerID="8f97e87ff243fa618bcea3b25297bd69d3ac3c710e362f714b1b5637249c17c3"
Oct 30 17:36:01.502874 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.502848    4433 generic.go:356] "Generic (PLEG): container finished" podID="7741e951-b0c5-432a-ae92-8dc492b47cf7" containerID="0cc3129a188de66ea5439687f2cb29905f6ac21823ed69b5a1a325291a4060e9" exitCode=0
Oct 30 17:36:01.503108 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.503085    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d4f99-8ff54d7d7-qp8mh" event={"ID":"7741e951-b0c5-432a-ae92-8dc492b47cf7","Type":"ContainerDied","Data":"0cc3129a188de66ea5439687f2cb29905f6ac21823ed69b5a1a325291a4060e9"}
Oct 30 17:36:01.515577 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.515550    4433 scope.go:122] "RemoveContainer" containerID="8f97e87ff243fa618bcea3b25297bd69d3ac3c710e362f714b1b5637249c17c3"
Oct 30 17:36:01.515949 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:36:01.515923    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f97e87ff243fa618bcea3b25297bd69d3ac3c710e362f714b1b5637249c17c3\": container with ID starting with 8f97e87ff243fa618bcea3b25297bd69d3ac3c710e362f714b1b5637249c17c3 not found: ID does not exist" containerID="8f97e87ff243fa618bcea3b25297bd69d3ac3c710e362f714b1b5637249c17c3"
Oct 30 17:36:01.516057 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.515960    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f97e87ff243fa618bcea3b25297bd69d3ac3c710e362f714b1b5637249c17c3"} err="failed to get container status \"8f97e87ff243fa618bcea3b25297bd69d3ac3c710e362f714b1b5637249c17c3\": rpc error: code = NotFound desc = could not find container \"8f97e87ff243fa618bcea3b25297bd69d3ac3c710e362f714b1b5637249c17c3\": container with ID starting with 8f97e87ff243fa618bcea3b25297bd69d3ac3c710e362f714b1b5637249c17c3 not found: ID does not exist"
Oct 30 17:36:01.550261 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.550223    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d4f99-8ff54d7d7-qp8mh"
Oct 30 17:36:01.553275 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.553246    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"]
Oct 30 17:36:01.563248 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.563220    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8681b5-openshift-default-7c5c845d8f-n8njg"]
Oct 30 17:36:01.640770 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.640726    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7741e951-b0c5-432a-ae92-8dc492b47cf7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7741e951-b0c5-432a-ae92-8dc492b47cf7-plugins-config-volume\") pod \"7741e951-b0c5-432a-ae92-8dc492b47cf7\" (UID: \"7741e951-b0c5-432a-ae92-8dc492b47cf7\") "
Oct 30 17:36:01.640956 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.640860    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7741e951-b0c5-432a-ae92-8dc492b47cf7-kube-api-access-b2lcq\" (UniqueName: \"kubernetes.io/projected/7741e951-b0c5-432a-ae92-8dc492b47cf7-kube-api-access-b2lcq\") pod \"7741e951-b0c5-432a-ae92-8dc492b47cf7\" (UID: \"7741e951-b0c5-432a-ae92-8dc492b47cf7\") "
Oct 30 17:36:01.641164 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.641142    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7741e951-b0c5-432a-ae92-8dc492b47cf7-plugins-config-volume" pod "7741e951-b0c5-432a-ae92-8dc492b47cf7" (UID: "7741e951-b0c5-432a-ae92-8dc492b47cf7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:36:01.641458 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.641433    4433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7741e951-b0c5-432a-ae92-8dc492b47cf7-plugins-config-volume\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:01.643046 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.643024    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7741e951-b0c5-432a-ae92-8dc492b47cf7-kube-api-access-b2lcq" pod "7741e951-b0c5-432a-ae92-8dc492b47cf7" (UID: "7741e951-b0c5-432a-ae92-8dc492b47cf7"). InnerVolumeSpecName "kube-api-access-b2lcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:36:01.696462 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.696390    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="303728db-2515-4854-a0ca-e8726a6414f5" path="/var/lib/kubelet/pods/303728db-2515-4854-a0ca-e8726a6414f5/volumes"
Oct 30 17:36:01.743006 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:01.742967    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2lcq\" (UniqueName: \"kubernetes.io/projected/7741e951-b0c5-432a-ae92-8dc492b47cf7-kube-api-access-b2lcq\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:02.510680 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:02.510644    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d4f99-8ff54d7d7-qp8mh"
Oct 30 17:36:02.511182 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:02.510682    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d4f99-8ff54d7d7-qp8mh" event={"ID":"7741e951-b0c5-432a-ae92-8dc492b47cf7","Type":"ContainerDied","Data":"8dbe06d8c141ca60da97a3dd4889b49294af69c91f1756abc1eac3af0cb78b2f"}
Oct 30 17:36:02.511182 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:02.510742    4433 scope.go:122] "RemoveContainer" containerID="0cc3129a188de66ea5439687f2cb29905f6ac21823ed69b5a1a325291a4060e9"
Oct 30 17:36:02.562773 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:02.562734    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d4f99-8ff54d7d7-qp8mh"]
Oct 30 17:36:02.574019 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:02.573986    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6d4f99-8ff54d7d7-qp8mh"]
Oct 30 17:36:03.653136 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:03.653101    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"]
Oct 30 17:36:03.653586 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:03.653324    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb" podUID="e4f0b0a7-7c36-4ad7-8573-582828ad701b" containerName="istio-proxy" containerID="cri-o://4f88f5602331c906a35bb13353dbf6a9a11b9fc3ee8ca48e3da5a76ba30ddd3f" gracePeriod=30
Oct 30 17:36:03.777212 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:03.776690    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7741e951-b0c5-432a-ae92-8dc492b47cf7" path="/var/lib/kubelet/pods/7741e951-b0c5-432a-ae92-8dc492b47cf7/volumes"
Oct 30 17:36:06.150104 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:06.150064    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:36:06.150529 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:06.150140    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb" podUID="e4f0b0a7-7c36-4ad7-8573-582828ad701b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:36:08.932719 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:08.932687    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:36:09.072714 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.072673    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4f0b0a7-7c36-4ad7-8573-582828ad701b-istio-token\" (UniqueName: \"kubernetes.io/projected/e4f0b0a7-7c36-4ad7-8573-582828ad701b-istio-token\") pod \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") "
Oct 30 17:36:09.072902 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.072724    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4f0b0a7-7c36-4ad7-8573-582828ad701b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4f0b0a7-7c36-4ad7-8573-582828ad701b-workload-socket\") pod \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") "
Oct 30 17:36:09.072902 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.072749    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4f0b0a7-7c36-4ad7-8573-582828ad701b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4f0b0a7-7c36-4ad7-8573-582828ad701b-workload-certs\") pod \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") "
Oct 30 17:36:09.072902 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.072807    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4f0b0a7-7c36-4ad7-8573-582828ad701b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4f0b0a7-7c36-4ad7-8573-582828ad701b-istio-podinfo\") pod \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") "
Oct 30 17:36:09.072902 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.072864    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4f0b0a7-7c36-4ad7-8573-582828ad701b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4f0b0a7-7c36-4ad7-8573-582828ad701b-istiod-ca-cert\") pod \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") "
Oct 30 17:36:09.073177 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.072913    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4f0b0a7-7c36-4ad7-8573-582828ad701b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4f0b0a7-7c36-4ad7-8573-582828ad701b-credential-socket\") pod \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") "
Oct 30 17:36:09.073177 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.072998    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4f0b0a7-7c36-4ad7-8573-582828ad701b-kube-api-access-2lvql\" (UniqueName: \"kubernetes.io/projected/e4f0b0a7-7c36-4ad7-8573-582828ad701b-kube-api-access-2lvql\") pod \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") "
Oct 30 17:36:09.073177 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.073022    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4f0b0a7-7c36-4ad7-8573-582828ad701b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4f0b0a7-7c36-4ad7-8573-582828ad701b-istio-envoy\") pod \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") "
Oct 30 17:36:09.073177 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.073053    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4f0b0a7-7c36-4ad7-8573-582828ad701b-workload-socket" pod "e4f0b0a7-7c36-4ad7-8573-582828ad701b" (UID: "e4f0b0a7-7c36-4ad7-8573-582828ad701b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:09.073177 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.073069    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4f0b0a7-7c36-4ad7-8573-582828ad701b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4f0b0a7-7c36-4ad7-8573-582828ad701b-istio-data\") pod \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\" (UID: \"e4f0b0a7-7c36-4ad7-8573-582828ad701b\") "
Oct 30 17:36:09.073491 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.073314    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4f0b0a7-7c36-4ad7-8573-582828ad701b-istio-data" pod "e4f0b0a7-7c36-4ad7-8573-582828ad701b" (UID: "e4f0b0a7-7c36-4ad7-8573-582828ad701b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:09.073701 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.073599    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4f0b0a7-7c36-4ad7-8573-582828ad701b-credential-socket" pod "e4f0b0a7-7c36-4ad7-8573-582828ad701b" (UID: "e4f0b0a7-7c36-4ad7-8573-582828ad701b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:09.073701 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.073655    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4f0b0a7-7c36-4ad7-8573-582828ad701b-istiod-ca-cert" pod "e4f0b0a7-7c36-4ad7-8573-582828ad701b" (UID: "e4f0b0a7-7c36-4ad7-8573-582828ad701b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:36:09.073919 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.073803    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4f0b0a7-7c36-4ad7-8573-582828ad701b-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:09.073919 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.073824    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4f0b0a7-7c36-4ad7-8573-582828ad701b-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:09.073919 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.073839    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4f0b0a7-7c36-4ad7-8573-582828ad701b-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:09.073919 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.073853    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4f0b0a7-7c36-4ad7-8573-582828ad701b-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:09.074427 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.074376    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4f0b0a7-7c36-4ad7-8573-582828ad701b-workload-certs" pod "e4f0b0a7-7c36-4ad7-8573-582828ad701b" (UID: "e4f0b0a7-7c36-4ad7-8573-582828ad701b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:09.075835 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.075807    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4f0b0a7-7c36-4ad7-8573-582828ad701b-kube-api-access-2lvql" pod "e4f0b0a7-7c36-4ad7-8573-582828ad701b" (UID: "e4f0b0a7-7c36-4ad7-8573-582828ad701b"). InnerVolumeSpecName "kube-api-access-2lvql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:36:09.075924 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.075892    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4f0b0a7-7c36-4ad7-8573-582828ad701b-istio-token" pod "e4f0b0a7-7c36-4ad7-8573-582828ad701b" (UID: "e4f0b0a7-7c36-4ad7-8573-582828ad701b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:36:09.075993 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.075975    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4f0b0a7-7c36-4ad7-8573-582828ad701b-istio-envoy" pod "e4f0b0a7-7c36-4ad7-8573-582828ad701b" (UID: "e4f0b0a7-7c36-4ad7-8573-582828ad701b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:09.076305 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.076286    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4f0b0a7-7c36-4ad7-8573-582828ad701b-istio-podinfo" pod "e4f0b0a7-7c36-4ad7-8573-582828ad701b" (UID: "e4f0b0a7-7c36-4ad7-8573-582828ad701b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:36:09.102605 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.102564    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"]
Oct 30 17:36:09.109583 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.109543    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"]
Oct 30 17:36:09.174807 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.174773    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4f0b0a7-7c36-4ad7-8573-582828ad701b-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:09.174807 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.174807    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4f0b0a7-7c36-4ad7-8573-582828ad701b-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:09.175032 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.174822    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4f0b0a7-7c36-4ad7-8573-582828ad701b-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:09.175032 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.174835    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lvql\" (UniqueName: \"kubernetes.io/projected/e4f0b0a7-7c36-4ad7-8573-582828ad701b-kube-api-access-2lvql\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:09.175032 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.174848    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4f0b0a7-7c36-4ad7-8573-582828ad701b-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:09.580118 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.580085    4433 generic.go:356] "Generic (PLEG): container finished" podID="e4f0b0a7-7c36-4ad7-8573-582828ad701b" containerID="4f88f5602331c906a35bb13353dbf6a9a11b9fc3ee8ca48e3da5a76ba30ddd3f" exitCode=0
Oct 30 17:36:09.580296 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.580149    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"
Oct 30 17:36:09.580296 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.580165    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb" event={"ID":"e4f0b0a7-7c36-4ad7-8573-582828ad701b","Type":"ContainerDied","Data":"4f88f5602331c906a35bb13353dbf6a9a11b9fc3ee8ca48e3da5a76ba30ddd3f"}
Oct 30 17:36:09.580296 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.580224    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb" event={"ID":"e4f0b0a7-7c36-4ad7-8573-582828ad701b","Type":"ContainerDied","Data":"bb9dc68ef4fd15822bb1204d8e33cfc2dd8e815e8969d2ffee04b22a53eb8f8f"}
Oct 30 17:36:09.580296 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.580249    4433 scope.go:122] "RemoveContainer" containerID="4f88f5602331c906a35bb13353dbf6a9a11b9fc3ee8ca48e3da5a76ba30ddd3f"
Oct 30 17:36:09.591747 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.591729    4433 scope.go:122] "RemoveContainer" containerID="2c1e5d2ddff5a0807279bf736c587e972339e07ddfeb32c620483e4d929ffe24"
Oct 30 17:36:09.602479 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.602456    4433 scope.go:122] "RemoveContainer" containerID="4f88f5602331c906a35bb13353dbf6a9a11b9fc3ee8ca48e3da5a76ba30ddd3f"
Oct 30 17:36:09.602811 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:36:09.602785    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f88f5602331c906a35bb13353dbf6a9a11b9fc3ee8ca48e3da5a76ba30ddd3f\": container with ID starting with 4f88f5602331c906a35bb13353dbf6a9a11b9fc3ee8ca48e3da5a76ba30ddd3f not found: ID does not exist" containerID="4f88f5602331c906a35bb13353dbf6a9a11b9fc3ee8ca48e3da5a76ba30ddd3f"
Oct 30 17:36:09.602923 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.602819    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f88f5602331c906a35bb13353dbf6a9a11b9fc3ee8ca48e3da5a76ba30ddd3f"} err="failed to get container status \"4f88f5602331c906a35bb13353dbf6a9a11b9fc3ee8ca48e3da5a76ba30ddd3f\": rpc error: code = NotFound desc = could not find container \"4f88f5602331c906a35bb13353dbf6a9a11b9fc3ee8ca48e3da5a76ba30ddd3f\": container with ID starting with 4f88f5602331c906a35bb13353dbf6a9a11b9fc3ee8ca48e3da5a76ba30ddd3f not found: ID does not exist"
Oct 30 17:36:09.602923 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.602838    4433 scope.go:122] "RemoveContainer" containerID="2c1e5d2ddff5a0807279bf736c587e972339e07ddfeb32c620483e4d929ffe24"
Oct 30 17:36:09.603096 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:36:09.603080    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c1e5d2ddff5a0807279bf736c587e972339e07ddfeb32c620483e4d929ffe24\": container with ID starting with 2c1e5d2ddff5a0807279bf736c587e972339e07ddfeb32c620483e4d929ffe24 not found: ID does not exist" containerID="2c1e5d2ddff5a0807279bf736c587e972339e07ddfeb32c620483e4d929ffe24"
Oct 30 17:36:09.603137 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.603102    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c1e5d2ddff5a0807279bf736c587e972339e07ddfeb32c620483e4d929ffe24"} err="failed to get container status \"2c1e5d2ddff5a0807279bf736c587e972339e07ddfeb32c620483e4d929ffe24\": rpc error: code = NotFound desc = could not find container \"2c1e5d2ddff5a0807279bf736c587e972339e07ddfeb32c620483e4d929ffe24\": container with ID starting with 2c1e5d2ddff5a0807279bf736c587e972339e07ddfeb32c620483e4d929ffe24 not found: ID does not exist"
Oct 30 17:36:09.645715 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.645673    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"]
Oct 30 17:36:09.682944 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.682904    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-s4hbb"]
Oct 30 17:36:09.697720 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:09.697688    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4f0b0a7-7c36-4ad7-8573-582828ad701b" path="/var/lib/kubelet/pods/e4f0b0a7-7c36-4ad7-8573-582828ad701b/volumes"
Oct 30 17:36:10.903560 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:10.903512    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw"]
Oct 30 17:36:10.904059 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:10.903766    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw" podUID="85c6e2b8-220e-4493-9468-e15d18966ae5" containerName="maas-api" containerID="cri-o://d06ada848dfcb340357207d468747a840c1f111aabdb6fe557d62edf3e09adb3" gracePeriod=30
Oct 30 17:36:11.278432 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:11.275071    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw"
Oct 30 17:36:11.445568 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:11.444053    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85c6e2b8-220e-4493-9468-e15d18966ae5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85c6e2b8-220e-4493-9468-e15d18966ae5-openshift-service-ca\") pod \"85c6e2b8-220e-4493-9468-e15d18966ae5\" (UID: \"85c6e2b8-220e-4493-9468-e15d18966ae5\") "
Oct 30 17:36:11.445568 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:11.444121    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85c6e2b8-220e-4493-9468-e15d18966ae5-kube-api-access-bfd85\" (UniqueName: \"kubernetes.io/projected/85c6e2b8-220e-4493-9468-e15d18966ae5-kube-api-access-bfd85\") pod \"85c6e2b8-220e-4493-9468-e15d18966ae5\" (UID: \"85c6e2b8-220e-4493-9468-e15d18966ae5\") "
Oct 30 17:36:11.445568 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:11.444184    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85c6e2b8-220e-4493-9468-e15d18966ae5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85c6e2b8-220e-4493-9468-e15d18966ae5-maas-api-tls\") pod \"85c6e2b8-220e-4493-9468-e15d18966ae5\" (UID: \"85c6e2b8-220e-4493-9468-e15d18966ae5\") "
Oct 30 17:36:11.445568 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:11.444279    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85c6e2b8-220e-4493-9468-e15d18966ae5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/85c6e2b8-220e-4493-9468-e15d18966ae5-metrics-certs\") pod \"85c6e2b8-220e-4493-9468-e15d18966ae5\" (UID: \"85c6e2b8-220e-4493-9468-e15d18966ae5\") "
Oct 30 17:36:11.445568 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:11.445073    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85c6e2b8-220e-4493-9468-e15d18966ae5-openshift-service-ca" pod "85c6e2b8-220e-4493-9468-e15d18966ae5" (UID: "85c6e2b8-220e-4493-9468-e15d18966ae5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:36:11.447781 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:11.447755    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85c6e2b8-220e-4493-9468-e15d18966ae5-metrics-certs" pod "85c6e2b8-220e-4493-9468-e15d18966ae5" (UID: "85c6e2b8-220e-4493-9468-e15d18966ae5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:36:11.448771 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:11.448728    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85c6e2b8-220e-4493-9468-e15d18966ae5-maas-api-tls" pod "85c6e2b8-220e-4493-9468-e15d18966ae5" (UID: "85c6e2b8-220e-4493-9468-e15d18966ae5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:36:11.448892 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:11.448865    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85c6e2b8-220e-4493-9468-e15d18966ae5-kube-api-access-bfd85" pod "85c6e2b8-220e-4493-9468-e15d18966ae5" (UID: "85c6e2b8-220e-4493-9468-e15d18966ae5"). InnerVolumeSpecName "kube-api-access-bfd85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:36:11.546531 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:11.546491    4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85c6e2b8-220e-4493-9468-e15d18966ae5-metrics-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:11.546531 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:11.546531    4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85c6e2b8-220e-4493-9468-e15d18966ae5-openshift-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:11.546772 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:11.546547    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfd85\" (UniqueName: \"kubernetes.io/projected/85c6e2b8-220e-4493-9468-e15d18966ae5-kube-api-access-bfd85\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:11.546772 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:11.546561    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85c6e2b8-220e-4493-9468-e15d18966ae5-maas-api-tls\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:11.597069 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:11.597029    4433 generic.go:356] "Generic (PLEG): container finished" podID="85c6e2b8-220e-4493-9468-e15d18966ae5" containerID="d06ada848dfcb340357207d468747a840c1f111aabdb6fe557d62edf3e09adb3" exitCode=0
Oct 30 17:36:11.597246 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:11.597083    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw"
Oct 30 17:36:11.597246 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:11.597106    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw" event={"ID":"85c6e2b8-220e-4493-9468-e15d18966ae5","Type":"ContainerDied","Data":"d06ada848dfcb340357207d468747a840c1f111aabdb6fe557d62edf3e09adb3"}
Oct 30 17:36:11.597246 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:11.597170    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw" event={"ID":"85c6e2b8-220e-4493-9468-e15d18966ae5","Type":"ContainerDied","Data":"266bad1c14231d02f198cf79c0ea08dc26c3fd5ad5376b3e08e66aba51e1fa46"}
Oct 30 17:36:11.597246 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:11.597201    4433 scope.go:122] "RemoveContainer" containerID="d06ada848dfcb340357207d468747a840c1f111aabdb6fe557d62edf3e09adb3"
Oct 30 17:36:11.607341 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:11.607317    4433 scope.go:122] "RemoveContainer" containerID="d06ada848dfcb340357207d468747a840c1f111aabdb6fe557d62edf3e09adb3"
Oct 30 17:36:11.607694 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:36:11.607673    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d06ada848dfcb340357207d468747a840c1f111aabdb6fe557d62edf3e09adb3\": container with ID starting with d06ada848dfcb340357207d468747a840c1f111aabdb6fe557d62edf3e09adb3 not found: ID does not exist" containerID="d06ada848dfcb340357207d468747a840c1f111aabdb6fe557d62edf3e09adb3"
Oct 30 17:36:11.607762 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:11.607705    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d06ada848dfcb340357207d468747a840c1f111aabdb6fe557d62edf3e09adb3"} err="failed to get container status \"d06ada848dfcb340357207d468747a840c1f111aabdb6fe557d62edf3e09adb3\": rpc error: code = NotFound desc = could not find container \"d06ada848dfcb340357207d468747a840c1f111aabdb6fe557d62edf3e09adb3\": container with ID starting with d06ada848dfcb340357207d468747a840c1f111aabdb6fe557d62edf3e09adb3 not found: ID does not exist"
Oct 30 17:36:11.648223 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:11.648189    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw"]
Oct 30 17:36:11.666792 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:11.663751    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6d4f99-97975b57b-xhgqw"]
Oct 30 17:36:11.711475 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:11.702956    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85c6e2b8-220e-4493-9468-e15d18966ae5" path="/var/lib/kubelet/pods/85c6e2b8-220e-4493-9468-e15d18966ae5/volumes"
Oct 30 17:36:23.917251 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:23.917204    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"]
Oct 30 17:36:23.917798 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:23.917499    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5" podUID="af923931-b247-4175-b077-658fe2f17c65" containerName="istio-proxy" containerID="cri-o://22e37b96fecc40d49fac7da3c2b051ed42ac6996135c4a0cb012923fa23eddb9" gracePeriod=30
Oct 30 17:36:24.124421 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.124365    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main" containerID="cri-o://f2af00ad3cdc877e1f0f327f66c6d10fe472bda51a28d38c7e419df07872d902" gracePeriod=45
Oct 30 17:36:24.130795 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.130757    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm" podUID="bf43bf3d-f126-40a1-a24f-10214a85ceb8" containerName="main" containerID="cri-o://0c6dc9d532bddec98d599936fc232a76278fd05ba4be5b2bf00477cab597be0a" gracePeriod=45
Oct 30 17:36:24.439076 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.439048    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:36:24.442663 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.442636    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 17:36:24.564239 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.564150    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf43bf3d-f126-40a1-a24f-10214a85ceb8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf43bf3d-f126-40a1-a24f-10214a85ceb8-tmp-dir\") pod \"bf43bf3d-f126-40a1-a24f-10214a85ceb8\" (UID: \"bf43bf3d-f126-40a1-a24f-10214a85ceb8\") "
Oct 30 17:36:24.564239 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.564211    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf43bf3d-f126-40a1-a24f-10214a85ceb8-kube-api-access-mdk6d\" (UniqueName: \"kubernetes.io/projected/bf43bf3d-f126-40a1-a24f-10214a85ceb8-kube-api-access-mdk6d\") pod \"bf43bf3d-f126-40a1-a24f-10214a85ceb8\" (UID: \"bf43bf3d-f126-40a1-a24f-10214a85ceb8\") "
Oct 30 17:36:24.564523 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.564250    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a8d170a-6410-4e32-837d-8a09cc612c6c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a8d170a-6410-4e32-837d-8a09cc612c6c-model-cache\") pod \"5a8d170a-6410-4e32-837d-8a09cc612c6c\" (UID: \"5a8d170a-6410-4e32-837d-8a09cc612c6c\") "
Oct 30 17:36:24.564523 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.564440    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a8d170a-6410-4e32-837d-8a09cc612c6c-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a8d170a-6410-4e32-837d-8a09cc612c6c-tls-certs\") pod \"5a8d170a-6410-4e32-837d-8a09cc612c6c\" (UID: \"5a8d170a-6410-4e32-837d-8a09cc612c6c\") "
Oct 30 17:36:24.564654 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.564517    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a8d170a-6410-4e32-837d-8a09cc612c6c-model-cache" pod "5a8d170a-6410-4e32-837d-8a09cc612c6c" (UID: "5a8d170a-6410-4e32-837d-8a09cc612c6c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:24.564654 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.564504    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf43bf3d-f126-40a1-a24f-10214a85ceb8-home\" (UniqueName: \"kubernetes.io/empty-dir/bf43bf3d-f126-40a1-a24f-10214a85ceb8-home\") pod \"bf43bf3d-f126-40a1-a24f-10214a85ceb8\" (UID: \"bf43bf3d-f126-40a1-a24f-10214a85ceb8\") "
Oct 30 17:36:24.564654 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.564591    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf43bf3d-f126-40a1-a24f-10214a85ceb8-tmp-dir" pod "bf43bf3d-f126-40a1-a24f-10214a85ceb8" (UID: "bf43bf3d-f126-40a1-a24f-10214a85ceb8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:24.564654 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.564636    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a8d170a-6410-4e32-837d-8a09cc612c6c-kube-api-access-88wcc\" (UniqueName: \"kubernetes.io/projected/5a8d170a-6410-4e32-837d-8a09cc612c6c-kube-api-access-88wcc\") pod \"5a8d170a-6410-4e32-837d-8a09cc612c6c\" (UID: \"5a8d170a-6410-4e32-837d-8a09cc612c6c\") "
Oct 30 17:36:24.564931 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.564726    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a8d170a-6410-4e32-837d-8a09cc612c6c-home\" (UniqueName: \"kubernetes.io/empty-dir/5a8d170a-6410-4e32-837d-8a09cc612c6c-home\") pod \"5a8d170a-6410-4e32-837d-8a09cc612c6c\" (UID: \"5a8d170a-6410-4e32-837d-8a09cc612c6c\") "
Oct 30 17:36:24.564931 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.564756    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf43bf3d-f126-40a1-a24f-10214a85ceb8-tls-certs\" (UniqueName: \"kubernetes.io/secret/bf43bf3d-f126-40a1-a24f-10214a85ceb8-tls-certs\") pod \"bf43bf3d-f126-40a1-a24f-10214a85ceb8\" (UID: \"bf43bf3d-f126-40a1-a24f-10214a85ceb8\") "
Oct 30 17:36:24.564931 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.564800    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf43bf3d-f126-40a1-a24f-10214a85ceb8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf43bf3d-f126-40a1-a24f-10214a85ceb8-model-cache\") pod \"bf43bf3d-f126-40a1-a24f-10214a85ceb8\" (UID: \"bf43bf3d-f126-40a1-a24f-10214a85ceb8\") "
Oct 30 17:36:24.564931 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.564873    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a8d170a-6410-4e32-837d-8a09cc612c6c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a8d170a-6410-4e32-837d-8a09cc612c6c-dshm\") pod \"5a8d170a-6410-4e32-837d-8a09cc612c6c\" (UID: \"5a8d170a-6410-4e32-837d-8a09cc612c6c\") "
Oct 30 17:36:24.564931 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.564927    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf43bf3d-f126-40a1-a24f-10214a85ceb8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf43bf3d-f126-40a1-a24f-10214a85ceb8-dshm\") pod \"bf43bf3d-f126-40a1-a24f-10214a85ceb8\" (UID: \"bf43bf3d-f126-40a1-a24f-10214a85ceb8\") "
Oct 30 17:36:24.565179 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.564953    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a8d170a-6410-4e32-837d-8a09cc612c6c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a8d170a-6410-4e32-837d-8a09cc612c6c-tmp-dir\") pod \"5a8d170a-6410-4e32-837d-8a09cc612c6c\" (UID: \"5a8d170a-6410-4e32-837d-8a09cc612c6c\") "
Oct 30 17:36:24.565338 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.565306    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf43bf3d-f126-40a1-a24f-10214a85ceb8-home" pod "bf43bf3d-f126-40a1-a24f-10214a85ceb8" (UID: "bf43bf3d-f126-40a1-a24f-10214a85ceb8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:24.565717 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.565582    4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf43bf3d-f126-40a1-a24f-10214a85ceb8-tmp-dir\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:24.565845 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.565829    4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a8d170a-6410-4e32-837d-8a09cc612c6c-model-cache\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:24.565944 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.565725    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf43bf3d-f126-40a1-a24f-10214a85ceb8-model-cache" pod "bf43bf3d-f126-40a1-a24f-10214a85ceb8" (UID: "bf43bf3d-f126-40a1-a24f-10214a85ceb8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:24.566042 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.565771    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a8d170a-6410-4e32-837d-8a09cc612c6c-tmp-dir" pod "5a8d170a-6410-4e32-837d-8a09cc612c6c" (UID: "5a8d170a-6410-4e32-837d-8a09cc612c6c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:24.566137 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.565974    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a8d170a-6410-4e32-837d-8a09cc612c6c-home" pod "5a8d170a-6410-4e32-837d-8a09cc612c6c" (UID: "5a8d170a-6410-4e32-837d-8a09cc612c6c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:24.570886 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.570861    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf43bf3d-f126-40a1-a24f-10214a85ceb8-kube-api-access-mdk6d" pod "bf43bf3d-f126-40a1-a24f-10214a85ceb8" (UID: "bf43bf3d-f126-40a1-a24f-10214a85ceb8"). InnerVolumeSpecName "kube-api-access-mdk6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:36:24.570974 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.570895    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a8d170a-6410-4e32-837d-8a09cc612c6c-tls-certs" pod "5a8d170a-6410-4e32-837d-8a09cc612c6c" (UID: "5a8d170a-6410-4e32-837d-8a09cc612c6c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:36:24.570974 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.570921    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a8d170a-6410-4e32-837d-8a09cc612c6c-kube-api-access-88wcc" pod "5a8d170a-6410-4e32-837d-8a09cc612c6c" (UID: "5a8d170a-6410-4e32-837d-8a09cc612c6c"). InnerVolumeSpecName "kube-api-access-88wcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:36:24.571145 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.571085    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a8d170a-6410-4e32-837d-8a09cc612c6c-dshm" pod "5a8d170a-6410-4e32-837d-8a09cc612c6c" (UID: "5a8d170a-6410-4e32-837d-8a09cc612c6c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:24.571275 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.571257    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf43bf3d-f126-40a1-a24f-10214a85ceb8-tls-certs" pod "bf43bf3d-f126-40a1-a24f-10214a85ceb8" (UID: "bf43bf3d-f126-40a1-a24f-10214a85ceb8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:36:24.571316 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.571272    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf43bf3d-f126-40a1-a24f-10214a85ceb8-dshm" pod "bf43bf3d-f126-40a1-a24f-10214a85ceb8" (UID: "bf43bf3d-f126-40a1-a24f-10214a85ceb8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:24.668430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.667600    4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf43bf3d-f126-40a1-a24f-10214a85ceb8-dshm\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:24.668430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.667644    4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a8d170a-6410-4e32-837d-8a09cc612c6c-tmp-dir\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:24.668430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.667661    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdk6d\" (UniqueName: \"kubernetes.io/projected/bf43bf3d-f126-40a1-a24f-10214a85ceb8-kube-api-access-mdk6d\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:24.668430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.667675    4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a8d170a-6410-4e32-837d-8a09cc612c6c-tls-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:24.668430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.667688    4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf43bf3d-f126-40a1-a24f-10214a85ceb8-home\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:24.668430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.667699    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88wcc\" (UniqueName: \"kubernetes.io/projected/5a8d170a-6410-4e32-837d-8a09cc612c6c-kube-api-access-88wcc\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:24.668430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.667711    4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a8d170a-6410-4e32-837d-8a09cc612c6c-home\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:24.668430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.667722    4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf43bf3d-f126-40a1-a24f-10214a85ceb8-tls-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:24.668430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.667734    4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf43bf3d-f126-40a1-a24f-10214a85ceb8-model-cache\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:24.668430 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.667746    4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a8d170a-6410-4e32-837d-8a09cc612c6c-dshm\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:24.721760 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.721723    4433 generic.go:356] "Generic (PLEG): container finished" podID="bf43bf3d-f126-40a1-a24f-10214a85ceb8" containerID="0c6dc9d532bddec98d599936fc232a76278fd05ba4be5b2bf00477cab597be0a" exitCode=0
Oct 30 17:36:24.721948 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.721772    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"
Oct 30 17:36:24.721948 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.721809    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm" event={"ID":"bf43bf3d-f126-40a1-a24f-10214a85ceb8","Type":"ContainerDied","Data":"0c6dc9d532bddec98d599936fc232a76278fd05ba4be5b2bf00477cab597be0a"}
Oct 30 17:36:24.721948 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.721862    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm" event={"ID":"bf43bf3d-f126-40a1-a24f-10214a85ceb8","Type":"ContainerDied","Data":"61b3dea60fa2b1105c8e9a1b2f0427412bb090f98cb59bcdb12dda80ca58f5a4"}
Oct 30 17:36:24.721948 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.721886    4433 scope.go:122] "RemoveContainer" containerID="0c6dc9d532bddec98d599936fc232a76278fd05ba4be5b2bf00477cab597be0a"
Oct 30 17:36:24.723604 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.723560    4433 generic.go:356] "Generic (PLEG): container finished" podID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerID="f2af00ad3cdc877e1f0f327f66c6d10fe472bda51a28d38c7e419df07872d902" exitCode=0
Oct 30 17:36:24.723711 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.723614    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"
Oct 30 17:36:24.723711 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.723606    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" event={"ID":"5a8d170a-6410-4e32-837d-8a09cc612c6c","Type":"ContainerDied","Data":"f2af00ad3cdc877e1f0f327f66c6d10fe472bda51a28d38c7e419df07872d902"}
Oct 30 17:36:24.723811 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.723717    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5" event={"ID":"5a8d170a-6410-4e32-837d-8a09cc612c6c","Type":"ContainerDied","Data":"0c4f7b90b8f7210a398cc8857629b8971487810fd6ab5425fe5edf1925d26c1f"}
Oct 30 17:36:24.731960 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.731929    4433 scope.go:122] "RemoveContainer" containerID="0c6dc9d532bddec98d599936fc232a76278fd05ba4be5b2bf00477cab597be0a"
Oct 30 17:36:24.732335 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:36:24.732305    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c6dc9d532bddec98d599936fc232a76278fd05ba4be5b2bf00477cab597be0a\": container with ID starting with 0c6dc9d532bddec98d599936fc232a76278fd05ba4be5b2bf00477cab597be0a not found: ID does not exist" containerID="0c6dc9d532bddec98d599936fc232a76278fd05ba4be5b2bf00477cab597be0a"
Oct 30 17:36:24.732547 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.732349    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c6dc9d532bddec98d599936fc232a76278fd05ba4be5b2bf00477cab597be0a"} err="failed to get container status \"0c6dc9d532bddec98d599936fc232a76278fd05ba4be5b2bf00477cab597be0a\": rpc error: code = NotFound desc = could not find container \"0c6dc9d532bddec98d599936fc232a76278fd05ba4be5b2bf00477cab597be0a\": container with ID starting with 0c6dc9d532bddec98d599936fc232a76278fd05ba4be5b2bf00477cab597be0a not found: ID does not exist"
Oct 30 17:36:24.732547 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.732374    4433 scope.go:122] "RemoveContainer" containerID="f2af00ad3cdc877e1f0f327f66c6d10fe472bda51a28d38c7e419df07872d902"
Oct 30 17:36:24.742482 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.742456    4433 scope.go:122] "RemoveContainer" containerID="f2af00ad3cdc877e1f0f327f66c6d10fe472bda51a28d38c7e419df07872d902"
Oct 30 17:36:24.742831 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:36:24.742810    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2af00ad3cdc877e1f0f327f66c6d10fe472bda51a28d38c7e419df07872d902\": container with ID starting with f2af00ad3cdc877e1f0f327f66c6d10fe472bda51a28d38c7e419df07872d902 not found: ID does not exist" containerID="f2af00ad3cdc877e1f0f327f66c6d10fe472bda51a28d38c7e419df07872d902"
Oct 30 17:36:24.742877 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.742841    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2af00ad3cdc877e1f0f327f66c6d10fe472bda51a28d38c7e419df07872d902"} err="failed to get container status \"f2af00ad3cdc877e1f0f327f66c6d10fe472bda51a28d38c7e419df07872d902\": rpc error: code = NotFound desc = could not find container \"f2af00ad3cdc877e1f0f327f66c6d10fe472bda51a28d38c7e419df07872d902\": container with ID starting with f2af00ad3cdc877e1f0f327f66c6d10fe472bda51a28d38c7e419df07872d902 not found: ID does not exist"
Oct 30 17:36:24.771475 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.771441    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"]
Oct 30 17:36:24.775536 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.775514    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-04248e/premium-w1-04248e-kserve-7dcd547c46-g9hv5"]
Oct 30 17:36:24.807730 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.807701    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"]
Oct 30 17:36:24.814196 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:24.814168    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-04248e/simulator-w1-04248e-kserve-8dd89fdf5-fw9dm"]
Oct 30 17:36:25.182474 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:25.182430    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"]
Oct 30 17:36:25.696731 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:25.696698    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" path="/var/lib/kubelet/pods/5a8d170a-6410-4e32-837d-8a09cc612c6c/volumes"
Oct 30 17:36:25.697192 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:25.697177    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf43bf3d-f126-40a1-a24f-10214a85ceb8" path="/var/lib/kubelet/pods/bf43bf3d-f126-40a1-a24f-10214a85ceb8/volumes"
Oct 30 17:36:26.981246 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:36:26.981205    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:36:28.693870 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.693836    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-755a1b1c-vsptj"]
Oct 30 17:36:28.694414 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694383    4433 cpu_manager.go:414] "containerMap: removing container" podUID="ba03c7a7-48e4-4051-91d4-9a030790fa8a" containerName="main"
Oct 30 17:36:28.694469 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694415    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba03c7a7-48e4-4051-91d4-9a030790fa8a" containerName="main"
Oct 30 17:36:28.694469 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694430    4433 cpu_manager.go:414] "containerMap: removing container" podUID="34119e6e-443a-40df-a54a-0f03b4e34553" containerName="istio-proxy"
Oct 30 17:36:28.694469 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694435    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34119e6e-443a-40df-a54a-0f03b4e34553" containerName="istio-proxy"
Oct 30 17:36:28.694469 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694443    4433 cpu_manager.go:414] "containerMap: removing container" podUID="5fcae069-8f9f-47e5-8d3a-82c248cf3c31" containerName="istio-proxy"
Oct 30 17:36:28.694469 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694448    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fcae069-8f9f-47e5-8d3a-82c248cf3c31" containerName="istio-proxy"
Oct 30 17:36:28.694469 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694460    4433 cpu_manager.go:414] "containerMap: removing container" podUID="e4f0b0a7-7c36-4ad7-8573-582828ad701b" containerName="istio-proxy"
Oct 30 17:36:28.694469 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694467    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4f0b0a7-7c36-4ad7-8573-582828ad701b" containerName="istio-proxy"
Oct 30 17:36:28.694694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694482    4433 cpu_manager.go:414] "containerMap: removing container" podUID="5fcae069-8f9f-47e5-8d3a-82c248cf3c31" containerName="istio-proxy"
Oct 30 17:36:28.694694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694487    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fcae069-8f9f-47e5-8d3a-82c248cf3c31" containerName="istio-proxy"
Oct 30 17:36:28.694694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694493    4433 cpu_manager.go:414] "containerMap: removing container" podUID="7741e951-b0c5-432a-ae92-8dc492b47cf7" containerName="payload-pre-processing"
Oct 30 17:36:28.694694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694497    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7741e951-b0c5-432a-ae92-8dc492b47cf7" containerName="payload-pre-processing"
Oct 30 17:36:28.694694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694508    4433 cpu_manager.go:414] "containerMap: removing container" podUID="1b3d63c8-3b08-4283-a20a-7cfae44ae212" containerName="maas-api"
Oct 30 17:36:28.694694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694512    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b3d63c8-3b08-4283-a20a-7cfae44ae212" containerName="maas-api"
Oct 30 17:36:28.694694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694518    4433 cpu_manager.go:414] "containerMap: removing container" podUID="db411383-a64b-4614-b496-abe53f6cc5c0" containerName="payload-processing"
Oct 30 17:36:28.694694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694523    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db411383-a64b-4614-b496-abe53f6cc5c0" containerName="payload-processing"
Oct 30 17:36:28.694694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694531    4433 cpu_manager.go:414] "containerMap: removing container" podUID="367df923-d896-440c-8644-86fbc2f6edaf" containerName="istio-proxy"
Oct 30 17:36:28.694694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694536    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="367df923-d896-440c-8644-86fbc2f6edaf" containerName="istio-proxy"
Oct 30 17:36:28.694694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694545    4433 cpu_manager.go:414] "containerMap: removing container" podUID="303728db-2515-4854-a0ca-e8726a6414f5" containerName="istio-proxy"
Oct 30 17:36:28.694694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694552    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="303728db-2515-4854-a0ca-e8726a6414f5" containerName="istio-proxy"
Oct 30 17:36:28.694694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694564    4433 cpu_manager.go:414] "containerMap: removing container" podUID="5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9" containerName="istio-proxy"
Oct 30 17:36:28.694694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694569    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9" containerName="istio-proxy"
Oct 30 17:36:28.694694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694575    4433 cpu_manager.go:414] "containerMap: removing container" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main"
Oct 30 17:36:28.694694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694579    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main"
Oct 30 17:36:28.694694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694586    4433 cpu_manager.go:414] "containerMap: removing container" podUID="35956888-3e58-4794-8613-bf44931f5129" containerName="revoke-keys"
Oct 30 17:36:28.694694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694591    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35956888-3e58-4794-8613-bf44931f5129" containerName="revoke-keys"
Oct 30 17:36:28.694694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694599    4433 cpu_manager.go:414] "containerMap: removing container" podUID="0cd29788-1dd4-4f3a-910e-0c180daed229" containerName="istio-proxy"
Oct 30 17:36:28.694694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694603    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cd29788-1dd4-4f3a-910e-0c180daed229" containerName="istio-proxy"
Oct 30 17:36:28.694694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694609    4433 cpu_manager.go:414] "containerMap: removing container" podUID="bf43bf3d-f126-40a1-a24f-10214a85ceb8" containerName="main"
Oct 30 17:36:28.694694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694613    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf43bf3d-f126-40a1-a24f-10214a85ceb8" containerName="main"
Oct 30 17:36:28.694694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694620    4433 cpu_manager.go:414] "containerMap: removing container" podUID="946311cc-f655-4798-bce5-10ad7d73c9e6" containerName="istio-proxy"
Oct 30 17:36:28.694694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694629    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="946311cc-f655-4798-bce5-10ad7d73c9e6" containerName="istio-proxy"
Oct 30 17:36:28.694694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694638    4433 cpu_manager.go:414] "containerMap: removing container" podUID="85c6e2b8-220e-4493-9468-e15d18966ae5" containerName="maas-api"
Oct 30 17:36:28.694694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694643    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85c6e2b8-220e-4493-9468-e15d18966ae5" containerName="maas-api"
Oct 30 17:36:28.694694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694668    4433 cpu_manager.go:414] "containerMap: removing container" podUID="d99bf885-403c-47ec-9b74-9c4c57d0e3ef" containerName="istio-proxy"
Oct 30 17:36:28.694694 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694672    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d99bf885-403c-47ec-9b74-9c4c57d0e3ef" containerName="istio-proxy"
Oct 30 17:36:28.695834 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694751    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="34119e6e-443a-40df-a54a-0f03b4e34553" containerName="istio-proxy"
Oct 30 17:36:28.695834 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694760    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fcae069-8f9f-47e5-8d3a-82c248cf3c31" containerName="istio-proxy"
Oct 30 17:36:28.695834 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694768    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4f0b0a7-7c36-4ad7-8573-582828ad701b" containerName="istio-proxy"
Oct 30 17:36:28.695834 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694775    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fcae069-8f9f-47e5-8d3a-82c248cf3c31" containerName="istio-proxy"
Oct 30 17:36:28.695834 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694780    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="7741e951-b0c5-432a-ae92-8dc492b47cf7" containerName="payload-pre-processing"
Oct 30 17:36:28.695834 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694785    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b3d63c8-3b08-4283-a20a-7cfae44ae212" containerName="maas-api"
Oct 30 17:36:28.695834 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694792    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="db411383-a64b-4614-b496-abe53f6cc5c0" containerName="payload-processing"
Oct 30 17:36:28.695834 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694799    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="367df923-d896-440c-8644-86fbc2f6edaf" containerName="istio-proxy"
Oct 30 17:36:28.695834 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694804    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="303728db-2515-4854-a0ca-e8726a6414f5" containerName="istio-proxy"
Oct 30 17:36:28.695834 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694810    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ce54e5a-f0d0-4f76-9f66-ad577aca3fe9" containerName="istio-proxy"
Oct 30 17:36:28.695834 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694815    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a8d170a-6410-4e32-837d-8a09cc612c6c" containerName="main"
Oct 30 17:36:28.695834 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694819    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4f0b0a7-7c36-4ad7-8573-582828ad701b" containerName="istio-proxy"
Oct 30 17:36:28.695834 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694825    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="35956888-3e58-4794-8613-bf44931f5129" containerName="revoke-keys"
Oct 30 17:36:28.695834 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694830    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cd29788-1dd4-4f3a-910e-0c180daed229" containerName="istio-proxy"
Oct 30 17:36:28.695834 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694835    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf43bf3d-f126-40a1-a24f-10214a85ceb8" containerName="main"
Oct 30 17:36:28.695834 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694839    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="946311cc-f655-4798-bce5-10ad7d73c9e6" containerName="istio-proxy"
Oct 30 17:36:28.695834 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694843    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="85c6e2b8-220e-4493-9468-e15d18966ae5" containerName="maas-api"
Oct 30 17:36:28.695834 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694853    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cd29788-1dd4-4f3a-910e-0c180daed229" containerName="istio-proxy"
Oct 30 17:36:28.695834 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694858    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="d99bf885-403c-47ec-9b74-9c4c57d0e3ef" containerName="istio-proxy"
Oct 30 17:36:28.695834 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.694866    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba03c7a7-48e4-4051-91d4-9a030790fa8a" containerName="main"
Oct 30 17:36:28.700183 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.700158    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755a1b1c-vsptj"
Oct 30 17:36:28.719104 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.719076    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-755a1b1c-vsptj"]
Oct 30 17:36:28.726048 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.726018    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:36:28.726197 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.726102    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\""
Oct 30 17:36:28.845997 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.845959    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8ae8154-7aa8-45be-80f8-3b53bd2a6842-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-755a1b1c-vsptj\" (UID: \"b8ae8154-7aa8-45be-80f8-3b53bd2a6842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755a1b1c-vsptj"
Oct 30 17:36:28.846171 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.846076    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8ae8154-7aa8-45be-80f8-3b53bd2a6842-maas-api-xks-ca\") pod \"maas-api-revoke-sub-755a1b1c-vsptj\" (UID: \"b8ae8154-7aa8-45be-80f8-3b53bd2a6842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755a1b1c-vsptj"
Oct 30 17:36:28.846171 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.846155    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nft49\" (UniqueName: \"kubernetes.io/projected/b8ae8154-7aa8-45be-80f8-3b53bd2a6842-kube-api-access-nft49\") pod \"maas-api-revoke-sub-755a1b1c-vsptj\" (UID: \"b8ae8154-7aa8-45be-80f8-3b53bd2a6842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755a1b1c-vsptj"
Oct 30 17:36:28.947257 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.947191    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nft49\" (UniqueName: \"kubernetes.io/projected/b8ae8154-7aa8-45be-80f8-3b53bd2a6842-kube-api-access-nft49\") pod \"maas-api-revoke-sub-755a1b1c-vsptj\" (UID: \"b8ae8154-7aa8-45be-80f8-3b53bd2a6842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755a1b1c-vsptj"
Oct 30 17:36:28.947386 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.947347    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8ae8154-7aa8-45be-80f8-3b53bd2a6842-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-755a1b1c-vsptj\" (UID: \"b8ae8154-7aa8-45be-80f8-3b53bd2a6842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755a1b1c-vsptj"
Oct 30 17:36:28.947496 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.947481    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8ae8154-7aa8-45be-80f8-3b53bd2a6842-maas-api-xks-ca\") pod \"maas-api-revoke-sub-755a1b1c-vsptj\" (UID: \"b8ae8154-7aa8-45be-80f8-3b53bd2a6842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755a1b1c-vsptj"
Oct 30 17:36:28.948054 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.948026    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8ae8154-7aa8-45be-80f8-3b53bd2a6842-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-755a1b1c-vsptj\" (UID: \"b8ae8154-7aa8-45be-80f8-3b53bd2a6842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755a1b1c-vsptj"
Oct 30 17:36:28.949773 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.949755    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8ae8154-7aa8-45be-80f8-3b53bd2a6842-maas-api-xks-ca\") pod \"maas-api-revoke-sub-755a1b1c-vsptj\" (UID: \"b8ae8154-7aa8-45be-80f8-3b53bd2a6842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755a1b1c-vsptj"
Oct 30 17:36:28.970012 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:28.969985    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nft49\" (UniqueName: \"kubernetes.io/projected/b8ae8154-7aa8-45be-80f8-3b53bd2a6842-kube-api-access-nft49\") pod \"maas-api-revoke-sub-755a1b1c-vsptj\" (UID: \"b8ae8154-7aa8-45be-80f8-3b53bd2a6842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755a1b1c-vsptj"
Oct 30 17:36:29.018025 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.017984    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755a1b1c-vsptj"
Oct 30 17:36:29.250816 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.250787    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"
Oct 30 17:36:29.344765 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.344111    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-755a1b1c-vsptj"]
Oct 30 17:36:29.346933 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:36:29.346898    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8ae8154_7aa8_45be_80f8_3b53bd2a6842.slice/crio-b9f08eed275833fc4afe64582ebd2d0a6175560e9e823adcd7d2d9a43b216578 WatchSource:0}: Error finding container b9f08eed275833fc4afe64582ebd2d0a6175560e9e823adcd7d2d9a43b216578: Status 404 returned error can't find the container with id b9f08eed275833fc4afe64582ebd2d0a6175560e9e823adcd7d2d9a43b216578
Oct 30 17:36:29.351931 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.351906    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af923931-b247-4175-b077-658fe2f17c65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af923931-b247-4175-b077-658fe2f17c65-istio-data\") pod \"af923931-b247-4175-b077-658fe2f17c65\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") "
Oct 30 17:36:29.352041 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.351948    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af923931-b247-4175-b077-658fe2f17c65-kube-api-access-m4q9v\" (UniqueName: \"kubernetes.io/projected/af923931-b247-4175-b077-658fe2f17c65-kube-api-access-m4q9v\") pod \"af923931-b247-4175-b077-658fe2f17c65\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") "
Oct 30 17:36:29.352041 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.351969    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af923931-b247-4175-b077-658fe2f17c65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af923931-b247-4175-b077-658fe2f17c65-istiod-ca-cert\") pod \"af923931-b247-4175-b077-658fe2f17c65\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") "
Oct 30 17:36:29.352041 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.352023    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af923931-b247-4175-b077-658fe2f17c65-istio-token\" (UniqueName: \"kubernetes.io/projected/af923931-b247-4175-b077-658fe2f17c65-istio-token\") pod \"af923931-b247-4175-b077-658fe2f17c65\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") "
Oct 30 17:36:29.352216 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.352042    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af923931-b247-4175-b077-658fe2f17c65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af923931-b247-4175-b077-658fe2f17c65-istio-envoy\") pod \"af923931-b247-4175-b077-658fe2f17c65\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") "
Oct 30 17:36:29.352216 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.352124    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af923931-b247-4175-b077-658fe2f17c65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af923931-b247-4175-b077-658fe2f17c65-workload-certs\") pod \"af923931-b247-4175-b077-658fe2f17c65\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") "
Oct 30 17:36:29.352216 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.352184    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af923931-b247-4175-b077-658fe2f17c65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af923931-b247-4175-b077-658fe2f17c65-workload-socket\") pod \"af923931-b247-4175-b077-658fe2f17c65\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") "
Oct 30 17:36:29.352216 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.352190    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af923931-b247-4175-b077-658fe2f17c65-istio-data" pod "af923931-b247-4175-b077-658fe2f17c65" (UID: "af923931-b247-4175-b077-658fe2f17c65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:29.352216 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.352212    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af923931-b247-4175-b077-658fe2f17c65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af923931-b247-4175-b077-658fe2f17c65-istio-podinfo\") pod \"af923931-b247-4175-b077-658fe2f17c65\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") "
Oct 30 17:36:29.352528 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.352258    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af923931-b247-4175-b077-658fe2f17c65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af923931-b247-4175-b077-658fe2f17c65-credential-socket\") pod \"af923931-b247-4175-b077-658fe2f17c65\" (UID: \"af923931-b247-4175-b077-658fe2f17c65\") "
Oct 30 17:36:29.352528 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.352453    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af923931-b247-4175-b077-658fe2f17c65-istiod-ca-cert" pod "af923931-b247-4175-b077-658fe2f17c65" (UID: "af923931-b247-4175-b077-658fe2f17c65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:36:29.352528 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.352468    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af923931-b247-4175-b077-658fe2f17c65-workload-certs" pod "af923931-b247-4175-b077-658fe2f17c65" (UID: "af923931-b247-4175-b077-658fe2f17c65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:29.352698 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.352607    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af923931-b247-4175-b077-658fe2f17c65-workload-socket" pod "af923931-b247-4175-b077-658fe2f17c65" (UID: "af923931-b247-4175-b077-658fe2f17c65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:29.352884 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.352861    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af923931-b247-4175-b077-658fe2f17c65-credential-socket" pod "af923931-b247-4175-b077-658fe2f17c65" (UID: "af923931-b247-4175-b077-658fe2f17c65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:29.353017 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.352995    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af923931-b247-4175-b077-658fe2f17c65-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:29.353087 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.353021    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af923931-b247-4175-b077-658fe2f17c65-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:29.353087 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.353035    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af923931-b247-4175-b077-658fe2f17c65-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:29.353087 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.353049    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af923931-b247-4175-b077-658fe2f17c65-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:29.353087 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.353074    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af923931-b247-4175-b077-658fe2f17c65-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:29.354337 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.354315    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af923931-b247-4175-b077-658fe2f17c65-istio-token" pod "af923931-b247-4175-b077-658fe2f17c65" (UID: "af923931-b247-4175-b077-658fe2f17c65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:36:29.354580 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.354552    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af923931-b247-4175-b077-658fe2f17c65-istio-envoy" pod "af923931-b247-4175-b077-658fe2f17c65" (UID: "af923931-b247-4175-b077-658fe2f17c65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:29.354696 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.354674    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af923931-b247-4175-b077-658fe2f17c65-kube-api-access-m4q9v" pod "af923931-b247-4175-b077-658fe2f17c65" (UID: "af923931-b247-4175-b077-658fe2f17c65"). InnerVolumeSpecName "kube-api-access-m4q9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:36:29.354792 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.354777    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af923931-b247-4175-b077-658fe2f17c65-istio-podinfo" pod "af923931-b247-4175-b077-658fe2f17c65" (UID: "af923931-b247-4175-b077-658fe2f17c65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:36:29.454701 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.454669    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4q9v\" (UniqueName: \"kubernetes.io/projected/af923931-b247-4175-b077-658fe2f17c65-kube-api-access-m4q9v\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:29.454701 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.454699    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af923931-b247-4175-b077-658fe2f17c65-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:29.454701 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.454708    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af923931-b247-4175-b077-658fe2f17c65-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:29.454936 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.454717    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af923931-b247-4175-b077-658fe2f17c65-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:29.759556 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.759516    4433 generic.go:356] "Generic (PLEG): container finished" podID="af923931-b247-4175-b077-658fe2f17c65" containerID="22e37b96fecc40d49fac7da3c2b051ed42ac6996135c4a0cb012923fa23eddb9" exitCode=0
Oct 30 17:36:29.760011 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.759560    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5" event={"ID":"af923931-b247-4175-b077-658fe2f17c65","Type":"ContainerDied","Data":"22e37b96fecc40d49fac7da3c2b051ed42ac6996135c4a0cb012923fa23eddb9"}
Oct 30 17:36:29.760011 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.759581    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"
Oct 30 17:36:29.760011 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.759622    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5" event={"ID":"af923931-b247-4175-b077-658fe2f17c65","Type":"ContainerDied","Data":"fb5c4187f29b3e3c0df9bfbaa5b6895b1317c5a2d2ddf5314bdc61c9e8b3e4e2"}
Oct 30 17:36:29.760011 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.759650    4433 scope.go:122] "RemoveContainer" containerID="22e37b96fecc40d49fac7da3c2b051ed42ac6996135c4a0cb012923fa23eddb9"
Oct 30 17:36:29.761873 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.761841    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755a1b1c-vsptj" event={"ID":"b8ae8154-7aa8-45be-80f8-3b53bd2a6842","Type":"ContainerStarted","Data":"03b7d814830ed0adae13182800f70bf07fe0f2ee2748bc46579d41d59cf09031"}
Oct 30 17:36:29.761987 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.761897    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755a1b1c-vsptj" event={"ID":"b8ae8154-7aa8-45be-80f8-3b53bd2a6842","Type":"ContainerStarted","Data":"b9f08eed275833fc4afe64582ebd2d0a6175560e9e823adcd7d2d9a43b216578"}
Oct 30 17:36:29.771278 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.771256    4433 scope.go:122] "RemoveContainer" containerID="22e37b96fecc40d49fac7da3c2b051ed42ac6996135c4a0cb012923fa23eddb9"
Oct 30 17:36:29.771629 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:36:29.771601    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22e37b96fecc40d49fac7da3c2b051ed42ac6996135c4a0cb012923fa23eddb9\": container with ID starting with 22e37b96fecc40d49fac7da3c2b051ed42ac6996135c4a0cb012923fa23eddb9 not found: ID does not exist" containerID="22e37b96fecc40d49fac7da3c2b051ed42ac6996135c4a0cb012923fa23eddb9"
Oct 30 17:36:29.771718 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.771652    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22e37b96fecc40d49fac7da3c2b051ed42ac6996135c4a0cb012923fa23eddb9"} err="failed to get container status \"22e37b96fecc40d49fac7da3c2b051ed42ac6996135c4a0cb012923fa23eddb9\": rpc error: code = NotFound desc = could not find container \"22e37b96fecc40d49fac7da3c2b051ed42ac6996135c4a0cb012923fa23eddb9\": container with ID starting with 22e37b96fecc40d49fac7da3c2b051ed42ac6996135c4a0cb012923fa23eddb9 not found: ID does not exist"
Oct 30 17:36:29.814015 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.813958    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755a1b1c-vsptj" podStartSLOduration=1.81394683 podStartE2EDuration="1.81394683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:29.813580361 +0000 UTC m=+2966.248482567" watchObservedRunningTime="2026-10-30 17:36:29.81394683 +0000 UTC m=+2966.248849036"
Oct 30 17:36:29.855724 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.855629    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"]
Oct 30 17:36:29.866580 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:29.866543    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-m7kb5"]
Oct 30 17:36:30.785015 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:30.782538    4433 generic.go:356] "Generic (PLEG): container finished" podID="b8ae8154-7aa8-45be-80f8-3b53bd2a6842" containerID="03b7d814830ed0adae13182800f70bf07fe0f2ee2748bc46579d41d59cf09031" exitCode=0
Oct 30 17:36:30.785015 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:30.782701    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755a1b1c-vsptj" event={"ID":"b8ae8154-7aa8-45be-80f8-3b53bd2a6842","Type":"ContainerDied","Data":"03b7d814830ed0adae13182800f70bf07fe0f2ee2748bc46579d41d59cf09031"}
Oct 30 17:36:31.702801 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:31.702760    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af923931-b247-4175-b077-658fe2f17c65" path="/var/lib/kubelet/pods/af923931-b247-4175-b077-658fe2f17c65/volumes"
Oct 30 17:36:31.942063 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:31.942036    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755a1b1c-vsptj"
Oct 30 17:36:31.990702 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:31.990611    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8ae8154-7aa8-45be-80f8-3b53bd2a6842-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8ae8154-7aa8-45be-80f8-3b53bd2a6842-maas-api-xks-ca\") pod \"b8ae8154-7aa8-45be-80f8-3b53bd2a6842\" (UID: \"b8ae8154-7aa8-45be-80f8-3b53bd2a6842\") "
Oct 30 17:36:31.990702 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:31.990666    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8ae8154-7aa8-45be-80f8-3b53bd2a6842-kube-api-access-nft49\" (UniqueName: \"kubernetes.io/projected/b8ae8154-7aa8-45be-80f8-3b53bd2a6842-kube-api-access-nft49\") pod \"b8ae8154-7aa8-45be-80f8-3b53bd2a6842\" (UID: \"b8ae8154-7aa8-45be-80f8-3b53bd2a6842\") "
Oct 30 17:36:31.990929 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:31.990711    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8ae8154-7aa8-45be-80f8-3b53bd2a6842-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8ae8154-7aa8-45be-80f8-3b53bd2a6842-maas-api-openshift-ca\") pod \"b8ae8154-7aa8-45be-80f8-3b53bd2a6842\" (UID: \"b8ae8154-7aa8-45be-80f8-3b53bd2a6842\") "
Oct 30 17:36:31.991450 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:31.991419    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8ae8154-7aa8-45be-80f8-3b53bd2a6842-maas-api-openshift-ca" pod "b8ae8154-7aa8-45be-80f8-3b53bd2a6842" (UID: "b8ae8154-7aa8-45be-80f8-3b53bd2a6842"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:36:31.999446 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:31.993337    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8ae8154-7aa8-45be-80f8-3b53bd2a6842-maas-api-xks-ca" pod "b8ae8154-7aa8-45be-80f8-3b53bd2a6842" (UID: "b8ae8154-7aa8-45be-80f8-3b53bd2a6842"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:36:31.999446 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:31.993772    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8ae8154-7aa8-45be-80f8-3b53bd2a6842-kube-api-access-nft49" pod "b8ae8154-7aa8-45be-80f8-3b53bd2a6842" (UID: "b8ae8154-7aa8-45be-80f8-3b53bd2a6842"). InnerVolumeSpecName "kube-api-access-nft49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:36:32.092555 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:32.092519    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8ae8154-7aa8-45be-80f8-3b53bd2a6842-maas-api-xks-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:32.092555 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:32.092554    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nft49\" (UniqueName: \"kubernetes.io/projected/b8ae8154-7aa8-45be-80f8-3b53bd2a6842-kube-api-access-nft49\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:32.092752 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:32.092567    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8ae8154-7aa8-45be-80f8-3b53bd2a6842-maas-api-openshift-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:32.801204 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:32.801172    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755a1b1c-vsptj"
Oct 30 17:36:32.801204 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:32.801197    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755a1b1c-vsptj" event={"ID":"b8ae8154-7aa8-45be-80f8-3b53bd2a6842","Type":"ContainerDied","Data":"b9f08eed275833fc4afe64582ebd2d0a6175560e9e823adcd7d2d9a43b216578"}
Oct 30 17:36:32.801480 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:32.801244    4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9f08eed275833fc4afe64582ebd2d0a6175560e9e823adcd7d2d9a43b216578"
Oct 30 17:36:34.880682 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:34.878723    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"]
Oct 30 17:36:34.884755 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:34.884714    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b" podUID="61e37504-76d8-47bf-8fa8-1a66d794e465" containerName="istio-proxy" containerID="cri-o://f27f2c87a1283ab0a1164f396e2868bdf2045780b19a1060ccae592e222ca276" gracePeriod=30
Oct 30 17:36:36.581378 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:36.581339    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:36:39.417846 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:39.417810    4433 patch_prober.go:29] interesting pod/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:36:39.418239 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:39.417864    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b" podUID="61e37504-76d8-47bf-8fa8-1a66d794e465" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:36:40.165632 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.165567    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"
Oct 30 17:36:40.205175 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.205111    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main" containerID="cri-o://ca5cf1b5f8831002e8775ce57802cebbd35efedc93b6f0e86c9a0936b07b9ae2" gracePeriod=45
Oct 30 17:36:40.357697 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.349938    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61e37504-76d8-47bf-8fa8-1a66d794e465-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61e37504-76d8-47bf-8fa8-1a66d794e465-istio-envoy\") pod \"61e37504-76d8-47bf-8fa8-1a66d794e465\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") "
Oct 30 17:36:40.357697 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.350007    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61e37504-76d8-47bf-8fa8-1a66d794e465-kube-api-access-ltftd\" (UniqueName: \"kubernetes.io/projected/61e37504-76d8-47bf-8fa8-1a66d794e465-kube-api-access-ltftd\") pod \"61e37504-76d8-47bf-8fa8-1a66d794e465\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") "
Oct 30 17:36:40.357697 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.350097    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61e37504-76d8-47bf-8fa8-1a66d794e465-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61e37504-76d8-47bf-8fa8-1a66d794e465-istio-podinfo\") pod \"61e37504-76d8-47bf-8fa8-1a66d794e465\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") "
Oct 30 17:36:40.357697 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.350150    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61e37504-76d8-47bf-8fa8-1a66d794e465-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61e37504-76d8-47bf-8fa8-1a66d794e465-istiod-ca-cert\") pod \"61e37504-76d8-47bf-8fa8-1a66d794e465\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") "
Oct 30 17:36:40.357697 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.350205    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61e37504-76d8-47bf-8fa8-1a66d794e465-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61e37504-76d8-47bf-8fa8-1a66d794e465-credential-socket\") pod \"61e37504-76d8-47bf-8fa8-1a66d794e465\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") "
Oct 30 17:36:40.357697 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.350296    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61e37504-76d8-47bf-8fa8-1a66d794e465-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61e37504-76d8-47bf-8fa8-1a66d794e465-workload-certs\") pod \"61e37504-76d8-47bf-8fa8-1a66d794e465\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") "
Oct 30 17:36:40.357697 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.350361    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61e37504-76d8-47bf-8fa8-1a66d794e465-istio-token\" (UniqueName: \"kubernetes.io/projected/61e37504-76d8-47bf-8fa8-1a66d794e465-istio-token\") pod \"61e37504-76d8-47bf-8fa8-1a66d794e465\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") "
Oct 30 17:36:40.357697 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.350661    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61e37504-76d8-47bf-8fa8-1a66d794e465-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61e37504-76d8-47bf-8fa8-1a66d794e465-istio-data\") pod \"61e37504-76d8-47bf-8fa8-1a66d794e465\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") "
Oct 30 17:36:40.357697 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.350844    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61e37504-76d8-47bf-8fa8-1a66d794e465-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61e37504-76d8-47bf-8fa8-1a66d794e465-workload-socket\") pod \"61e37504-76d8-47bf-8fa8-1a66d794e465\" (UID: \"61e37504-76d8-47bf-8fa8-1a66d794e465\") "
Oct 30 17:36:40.357697 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.352118    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61e37504-76d8-47bf-8fa8-1a66d794e465-workload-socket" pod "61e37504-76d8-47bf-8fa8-1a66d794e465" (UID: "61e37504-76d8-47bf-8fa8-1a66d794e465"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:40.357697 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.354729    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61e37504-76d8-47bf-8fa8-1a66d794e465-istiod-ca-cert" pod "61e37504-76d8-47bf-8fa8-1a66d794e465" (UID: "61e37504-76d8-47bf-8fa8-1a66d794e465"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:36:40.383455 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.376606    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61e37504-76d8-47bf-8fa8-1a66d794e465-credential-socket" pod "61e37504-76d8-47bf-8fa8-1a66d794e465" (UID: "61e37504-76d8-47bf-8fa8-1a66d794e465"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:40.402654 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.398506    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61e37504-76d8-47bf-8fa8-1a66d794e465-istio-data" pod "61e37504-76d8-47bf-8fa8-1a66d794e465" (UID: "61e37504-76d8-47bf-8fa8-1a66d794e465"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:40.402741 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.402649    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"]
Oct 30 17:36:40.432474 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.428729    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61e37504-76d8-47bf-8fa8-1a66d794e465-workload-certs" pod "61e37504-76d8-47bf-8fa8-1a66d794e465" (UID: "61e37504-76d8-47bf-8fa8-1a66d794e465"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:40.442551 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.434028    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61e37504-76d8-47bf-8fa8-1a66d794e465-kube-api-access-ltftd" pod "61e37504-76d8-47bf-8fa8-1a66d794e465" (UID: "61e37504-76d8-47bf-8fa8-1a66d794e465"). InnerVolumeSpecName "kube-api-access-ltftd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:36:40.442551 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.438614    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61e37504-76d8-47bf-8fa8-1a66d794e465-istio-podinfo" pod "61e37504-76d8-47bf-8fa8-1a66d794e465" (UID: "61e37504-76d8-47bf-8fa8-1a66d794e465"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:36:40.484428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.461600    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61e37504-76d8-47bf-8fa8-1a66d794e465-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:40.484428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.461647    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61e37504-76d8-47bf-8fa8-1a66d794e465-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:40.484428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.461663    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61e37504-76d8-47bf-8fa8-1a66d794e465-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:40.484428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.461677    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61e37504-76d8-47bf-8fa8-1a66d794e465-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:40.484428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.461692    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61e37504-76d8-47bf-8fa8-1a66d794e465-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:40.484428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.461704    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61e37504-76d8-47bf-8fa8-1a66d794e465-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:40.484428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.461716    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltftd\" (UniqueName: \"kubernetes.io/projected/61e37504-76d8-47bf-8fa8-1a66d794e465-kube-api-access-ltftd\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:40.498566 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.498502    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61e37504-76d8-47bf-8fa8-1a66d794e465-istio-token" pod "61e37504-76d8-47bf-8fa8-1a66d794e465" (UID: "61e37504-76d8-47bf-8fa8-1a66d794e465"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:36:40.498774 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.498582    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61e37504-76d8-47bf-8fa8-1a66d794e465-istio-envoy" pod "61e37504-76d8-47bf-8fa8-1a66d794e465" (UID: "61e37504-76d8-47bf-8fa8-1a66d794e465"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:40.567826 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.567723    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61e37504-76d8-47bf-8fa8-1a66d794e465-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:40.567826 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.567766    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61e37504-76d8-47bf-8fa8-1a66d794e465-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:40.597645 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.597615    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 17:36:40.770822 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.770718    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71a27c69-89cd-4802-a397-eff106d58bd1-tls-certs\" (UniqueName: \"kubernetes.io/secret/71a27c69-89cd-4802-a397-eff106d58bd1-tls-certs\") pod \"71a27c69-89cd-4802-a397-eff106d58bd1\" (UID: \"71a27c69-89cd-4802-a397-eff106d58bd1\") "
Oct 30 17:36:40.770822 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.770808    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a27c69-89cd-4802-a397-eff106d58bd1-home\" (UniqueName: \"kubernetes.io/empty-dir/71a27c69-89cd-4802-a397-eff106d58bd1-home\") pod \"71a27c69-89cd-4802-a397-eff106d58bd1\" (UID: \"71a27c69-89cd-4802-a397-eff106d58bd1\") "
Oct 30 17:36:40.771058 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.770973    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a27c69-89cd-4802-a397-eff106d58bd1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a27c69-89cd-4802-a397-eff106d58bd1-model-cache\") pod \"71a27c69-89cd-4802-a397-eff106d58bd1\" (UID: \"71a27c69-89cd-4802-a397-eff106d58bd1\") "
Oct 30 17:36:40.771058 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.771036    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a27c69-89cd-4802-a397-eff106d58bd1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a27c69-89cd-4802-a397-eff106d58bd1-tmp-dir\") pod \"71a27c69-89cd-4802-a397-eff106d58bd1\" (UID: \"71a27c69-89cd-4802-a397-eff106d58bd1\") "
Oct 30 17:36:40.771176 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.771071    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71a27c69-89cd-4802-a397-eff106d58bd1-kube-api-access-9p9c6\" (UniqueName: \"kubernetes.io/projected/71a27c69-89cd-4802-a397-eff106d58bd1-kube-api-access-9p9c6\") pod \"71a27c69-89cd-4802-a397-eff106d58bd1\" (UID: \"71a27c69-89cd-4802-a397-eff106d58bd1\") "
Oct 30 17:36:40.771176 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.771076    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a27c69-89cd-4802-a397-eff106d58bd1-home" pod "71a27c69-89cd-4802-a397-eff106d58bd1" (UID: "71a27c69-89cd-4802-a397-eff106d58bd1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:40.771176 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.771127    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a27c69-89cd-4802-a397-eff106d58bd1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a27c69-89cd-4802-a397-eff106d58bd1-dshm\") pod \"71a27c69-89cd-4802-a397-eff106d58bd1\" (UID: \"71a27c69-89cd-4802-a397-eff106d58bd1\") "
Oct 30 17:36:40.771348 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.771259    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a27c69-89cd-4802-a397-eff106d58bd1-model-cache" pod "71a27c69-89cd-4802-a397-eff106d58bd1" (UID: "71a27c69-89cd-4802-a397-eff106d58bd1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:40.771348 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.771286    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a27c69-89cd-4802-a397-eff106d58bd1-tmp-dir" pod "71a27c69-89cd-4802-a397-eff106d58bd1" (UID: "71a27c69-89cd-4802-a397-eff106d58bd1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:40.771727 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.771703    4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a27c69-89cd-4802-a397-eff106d58bd1-model-cache\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:40.771727 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.771727    4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a27c69-89cd-4802-a397-eff106d58bd1-tmp-dir\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:40.771952 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.771740    4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a27c69-89cd-4802-a397-eff106d58bd1-home\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:40.773376 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.773350    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71a27c69-89cd-4802-a397-eff106d58bd1-tls-certs" pod "71a27c69-89cd-4802-a397-eff106d58bd1" (UID: "71a27c69-89cd-4802-a397-eff106d58bd1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:36:40.773595 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.773569    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a27c69-89cd-4802-a397-eff106d58bd1-dshm" pod "71a27c69-89cd-4802-a397-eff106d58bd1" (UID: "71a27c69-89cd-4802-a397-eff106d58bd1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:40.773698 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.773677    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a27c69-89cd-4802-a397-eff106d58bd1-kube-api-access-9p9c6" pod "71a27c69-89cd-4802-a397-eff106d58bd1" (UID: "71a27c69-89cd-4802-a397-eff106d58bd1"). InnerVolumeSpecName "kube-api-access-9p9c6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:36:40.854647 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.854609    4433 generic.go:356] "Generic (PLEG): container finished" podID="71a27c69-89cd-4802-a397-eff106d58bd1" containerID="ca5cf1b5f8831002e8775ce57802cebbd35efedc93b6f0e86c9a0936b07b9ae2" exitCode=0
Oct 30 17:36:40.854857 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.854660    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"
Oct 30 17:36:40.854857 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.854694    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" event={"ID":"71a27c69-89cd-4802-a397-eff106d58bd1","Type":"ContainerDied","Data":"ca5cf1b5f8831002e8775ce57802cebbd35efedc93b6f0e86c9a0936b07b9ae2"}
Oct 30 17:36:40.854857 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.854746    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v" event={"ID":"71a27c69-89cd-4802-a397-eff106d58bd1","Type":"ContainerDied","Data":"c3fb3fa58cdf54cf5fb2952cf20b5526a111fef89172892ebf9fffbde40d61b6"}
Oct 30 17:36:40.854857 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.854767    4433 scope.go:122] "RemoveContainer" containerID="ca5cf1b5f8831002e8775ce57802cebbd35efedc93b6f0e86c9a0936b07b9ae2"
Oct 30 17:36:40.856558 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.856526    4433 generic.go:356] "Generic (PLEG): container finished" podID="61e37504-76d8-47bf-8fa8-1a66d794e465" containerID="f27f2c87a1283ab0a1164f396e2868bdf2045780b19a1060ccae592e222ca276" exitCode=0
Oct 30 17:36:40.856673 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.856597    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"
Oct 30 17:36:40.856673 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.856614    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b" event={"ID":"61e37504-76d8-47bf-8fa8-1a66d794e465","Type":"ContainerDied","Data":"f27f2c87a1283ab0a1164f396e2868bdf2045780b19a1060ccae592e222ca276"}
Oct 30 17:36:40.856754 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.856679    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b" event={"ID":"61e37504-76d8-47bf-8fa8-1a66d794e465","Type":"ContainerDied","Data":"4965466a76266e3e6d8ab3e3c8870a53ed9526ec0a2c81c0ddbfae8728e50d32"}
Oct 30 17:36:40.865862 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.865840    4433 scope.go:122] "RemoveContainer" containerID="ca5cf1b5f8831002e8775ce57802cebbd35efedc93b6f0e86c9a0936b07b9ae2"
Oct 30 17:36:40.866247 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:36:40.866214    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca5cf1b5f8831002e8775ce57802cebbd35efedc93b6f0e86c9a0936b07b9ae2\": container with ID starting with ca5cf1b5f8831002e8775ce57802cebbd35efedc93b6f0e86c9a0936b07b9ae2 not found: ID does not exist" containerID="ca5cf1b5f8831002e8775ce57802cebbd35efedc93b6f0e86c9a0936b07b9ae2"
Oct 30 17:36:40.866439 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.866259    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca5cf1b5f8831002e8775ce57802cebbd35efedc93b6f0e86c9a0936b07b9ae2"} err="failed to get container status \"ca5cf1b5f8831002e8775ce57802cebbd35efedc93b6f0e86c9a0936b07b9ae2\": rpc error: code = NotFound desc = could not find container \"ca5cf1b5f8831002e8775ce57802cebbd35efedc93b6f0e86c9a0936b07b9ae2\": container with ID starting with ca5cf1b5f8831002e8775ce57802cebbd35efedc93b6f0e86c9a0936b07b9ae2 not found: ID does not exist"
Oct 30 17:36:40.866439 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.866285    4433 scope.go:122] "RemoveContainer" containerID="f27f2c87a1283ab0a1164f396e2868bdf2045780b19a1060ccae592e222ca276"
Oct 30 17:36:40.872766 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.872734    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p9c6\" (UniqueName: \"kubernetes.io/projected/71a27c69-89cd-4802-a397-eff106d58bd1-kube-api-access-9p9c6\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:40.872766 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.872764    4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a27c69-89cd-4802-a397-eff106d58bd1-dshm\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:40.872944 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.872775    4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a27c69-89cd-4802-a397-eff106d58bd1-tls-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:40.877165 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.877140    4433 scope.go:122] "RemoveContainer" containerID="f27f2c87a1283ab0a1164f396e2868bdf2045780b19a1060ccae592e222ca276"
Oct 30 17:36:40.877502 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:36:40.877483    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f27f2c87a1283ab0a1164f396e2868bdf2045780b19a1060ccae592e222ca276\": container with ID starting with f27f2c87a1283ab0a1164f396e2868bdf2045780b19a1060ccae592e222ca276 not found: ID does not exist" containerID="f27f2c87a1283ab0a1164f396e2868bdf2045780b19a1060ccae592e222ca276"
Oct 30 17:36:40.877552 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.877511    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f27f2c87a1283ab0a1164f396e2868bdf2045780b19a1060ccae592e222ca276"} err="failed to get container status \"f27f2c87a1283ab0a1164f396e2868bdf2045780b19a1060ccae592e222ca276\": rpc error: code = NotFound desc = could not find container \"f27f2c87a1283ab0a1164f396e2868bdf2045780b19a1060ccae592e222ca276\": container with ID starting with f27f2c87a1283ab0a1164f396e2868bdf2045780b19a1060ccae592e222ca276 not found: ID does not exist"
Oct 30 17:36:40.914381 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.914348    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"]
Oct 30 17:36:40.918752 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.918715    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-2wp2b"]
Oct 30 17:36:40.952916 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.952874    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"]
Oct 30 17:36:40.956609 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:40.956564    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-b4f406/simulator-w2-b4f406-kserve-7fd7658bff-vpq4v"]
Oct 30 17:36:41.695008 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:41.694966    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61e37504-76d8-47bf-8fa8-1a66d794e465" path="/var/lib/kubelet/pods/61e37504-76d8-47bf-8fa8-1a66d794e465/volumes"
Oct 30 17:36:41.695638 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:41.695616    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" path="/var/lib/kubelet/pods/71a27c69-89cd-4802-a397-eff106d58bd1/volumes"
Oct 30 17:36:52.099477 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:52.099436    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ef70ed1-qlxbp"]
Oct 30 17:36:52.099971 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:52.099956    4433 cpu_manager.go:414] "containerMap: removing container" podUID="61e37504-76d8-47bf-8fa8-1a66d794e465" containerName="istio-proxy"
Oct 30 17:36:52.099971 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:52.099968    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61e37504-76d8-47bf-8fa8-1a66d794e465" containerName="istio-proxy"
Oct 30 17:36:52.100050 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:52.099989    4433 cpu_manager.go:414] "containerMap: removing container" podUID="b8ae8154-7aa8-45be-80f8-3b53bd2a6842" containerName="revoke-keys"
Oct 30 17:36:52.100050 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:52.099995    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8ae8154-7aa8-45be-80f8-3b53bd2a6842" containerName="revoke-keys"
Oct 30 17:36:52.100050 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:52.100004    4433 cpu_manager.go:414] "containerMap: removing container" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main"
Oct 30 17:36:52.100050 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:52.100010    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main"
Oct 30 17:36:52.100050 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:52.100018    4433 cpu_manager.go:414] "containerMap: removing container" podUID="34119e6e-443a-40df-a54a-0f03b4e34553" containerName="istio-proxy"
Oct 30 17:36:52.100050 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:52.100023    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34119e6e-443a-40df-a54a-0f03b4e34553" containerName="istio-proxy"
Oct 30 17:36:52.100050 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:52.100033    4433 cpu_manager.go:414] "containerMap: removing container" podUID="e4f0b0a7-7c36-4ad7-8573-582828ad701b" containerName="istio-proxy"
Oct 30 17:36:52.100050 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:52.100037    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4f0b0a7-7c36-4ad7-8573-582828ad701b" containerName="istio-proxy"
Oct 30 17:36:52.100050 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:52.100044    4433 cpu_manager.go:414] "containerMap: removing container" podUID="af923931-b247-4175-b077-658fe2f17c65" containerName="istio-proxy"
Oct 30 17:36:52.100050 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:52.100048    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af923931-b247-4175-b077-658fe2f17c65" containerName="istio-proxy"
Oct 30 17:36:52.100363 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:52.100066    4433 cpu_manager.go:414] "containerMap: removing container" podUID="0cd29788-1dd4-4f3a-910e-0c180daed229" containerName="istio-proxy"
Oct 30 17:36:52.100363 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:52.100071    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cd29788-1dd4-4f3a-910e-0c180daed229" containerName="istio-proxy"
Oct 30 17:36:52.100363 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:52.100157    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="61e37504-76d8-47bf-8fa8-1a66d794e465" containerName="istio-proxy"
Oct 30 17:36:52.100363 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:52.100171    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8ae8154-7aa8-45be-80f8-3b53bd2a6842" containerName="revoke-keys"
Oct 30 17:36:52.100363 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:52.100182    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="71a27c69-89cd-4802-a397-eff106d58bd1" containerName="main"
Oct 30 17:36:52.100363 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:52.100188    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="34119e6e-443a-40df-a54a-0f03b4e34553" containerName="istio-proxy"
Oct 30 17:36:52.100363 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:52.100194    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="af923931-b247-4175-b077-658fe2f17c65" containerName="istio-proxy"
Oct 30 17:36:52.105529 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:52.105504    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef70ed1-qlxbp"
Oct 30 17:36:52.127897 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:52.127864    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ef70ed1-qlxbp"]
Oct 30 17:36:52.199586 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:52.199550    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea0c2df3-91dd-40ae-8c3b-5588b8dc3517-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ef70ed1-qlxbp\" (UID: \"ea0c2df3-91dd-40ae-8c3b-5588b8dc3517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef70ed1-qlxbp"
Oct 30 17:36:52.199794 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:52.199726    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmssk\" (UniqueName: \"kubernetes.io/projected/ea0c2df3-91dd-40ae-8c3b-5588b8dc3517-kube-api-access-kmssk\") pod \"maas-api-revoke-sub-1ef70ed1-qlxbp\" (UID: \"ea0c2df3-91dd-40ae-8c3b-5588b8dc3517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef70ed1-qlxbp"
Oct 30 17:36:52.199913 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:52.199895    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea0c2df3-91dd-40ae-8c3b-5588b8dc3517-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ef70ed1-qlxbp\" (UID: \"ea0c2df3-91dd-40ae-8c3b-5588b8dc3517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef70ed1-qlxbp"
Oct 30 17:36:52.204047 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:52.204022    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:36:52.204173 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:52.204094    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\""
Oct 30 17:36:52.301297 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:52.301254    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea0c2df3-91dd-40ae-8c3b-5588b8dc3517-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ef70ed1-qlxbp\" (UID: \"ea0c2df3-91dd-40ae-8c3b-5588b8dc3517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef70ed1-qlxbp"
Oct 30 17:36:52.301539 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:52.301360    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea0c2df3-91dd-40ae-8c3b-5588b8dc3517-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ef70ed1-qlxbp\" (UID: \"ea0c2df3-91dd-40ae-8c3b-5588b8dc3517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef70ed1-qlxbp"
Oct 30 17:36:52.301539 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:52.301512    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmssk\" (UniqueName: \"kubernetes.io/projected/ea0c2df3-91dd-40ae-8c3b-5588b8dc3517-kube-api-access-kmssk\") pod \"maas-api-revoke-sub-1ef70ed1-qlxbp\" (UID: \"ea0c2df3-91dd-40ae-8c3b-5588b8dc3517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef70ed1-qlxbp"
Oct 30 17:36:52.302022 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:52.301989    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea0c2df3-91dd-40ae-8c3b-5588b8dc3517-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ef70ed1-qlxbp\" (UID: \"ea0c2df3-91dd-40ae-8c3b-5588b8dc3517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef70ed1-qlxbp"
Oct 30 17:36:52.303741 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:52.303718    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea0c2df3-91dd-40ae-8c3b-5588b8dc3517-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ef70ed1-qlxbp\" (UID: \"ea0c2df3-91dd-40ae-8c3b-5588b8dc3517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef70ed1-qlxbp"
Oct 30 17:36:52.325044 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:52.325009    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmssk\" (UniqueName: \"kubernetes.io/projected/ea0c2df3-91dd-40ae-8c3b-5588b8dc3517-kube-api-access-kmssk\") pod \"maas-api-revoke-sub-1ef70ed1-qlxbp\" (UID: \"ea0c2df3-91dd-40ae-8c3b-5588b8dc3517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef70ed1-qlxbp"
Oct 30 17:36:52.423981 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:52.423892    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef70ed1-qlxbp"
Oct 30 17:36:52.748220 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:52.748190    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ef70ed1-qlxbp"]
Oct 30 17:36:52.750591 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:36:52.750558    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea0c2df3_91dd_40ae_8c3b_5588b8dc3517.slice/crio-ba0f73f8e553b18336052caa41932ce18e5468430058e1082077daf304f9ddc0 WatchSource:0}: Error finding container ba0f73f8e553b18336052caa41932ce18e5468430058e1082077daf304f9ddc0: Status 404 returned error can't find the container with id ba0f73f8e553b18336052caa41932ce18e5468430058e1082077daf304f9ddc0
Oct 30 17:36:53.005001 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:53.004914    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef70ed1-qlxbp" event={"ID":"ea0c2df3-91dd-40ae-8c3b-5588b8dc3517","Type":"ContainerStarted","Data":"ba0f73f8e553b18336052caa41932ce18e5468430058e1082077daf304f9ddc0"}
Oct 30 17:36:54.012763 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:54.012728    4433 generic.go:356] "Generic (PLEG): container finished" podID="ea0c2df3-91dd-40ae-8c3b-5588b8dc3517" containerID="7867db29edcbd7a4aaefcc211a3affcf434a603d25ca38392f1c933ce18b61ff" exitCode=0
Oct 30 17:36:54.013172 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:54.012771    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef70ed1-qlxbp" event={"ID":"ea0c2df3-91dd-40ae-8c3b-5588b8dc3517","Type":"ContainerDied","Data":"7867db29edcbd7a4aaefcc211a3affcf434a603d25ca38392f1c933ce18b61ff"}
Oct 30 17:36:55.163640 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.163615    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef70ed1-qlxbp"
Oct 30 17:36:55.338172 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.338095    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea0c2df3-91dd-40ae-8c3b-5588b8dc3517-kube-api-access-kmssk\" (UniqueName: \"kubernetes.io/projected/ea0c2df3-91dd-40ae-8c3b-5588b8dc3517-kube-api-access-kmssk\") pod \"ea0c2df3-91dd-40ae-8c3b-5588b8dc3517\" (UID: \"ea0c2df3-91dd-40ae-8c3b-5588b8dc3517\") "
Oct 30 17:36:55.338172 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.338163    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea0c2df3-91dd-40ae-8c3b-5588b8dc3517-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea0c2df3-91dd-40ae-8c3b-5588b8dc3517-maas-api-xks-ca\") pod \"ea0c2df3-91dd-40ae-8c3b-5588b8dc3517\" (UID: \"ea0c2df3-91dd-40ae-8c3b-5588b8dc3517\") "
Oct 30 17:36:55.338414 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.338188    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea0c2df3-91dd-40ae-8c3b-5588b8dc3517-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea0c2df3-91dd-40ae-8c3b-5588b8dc3517-maas-api-openshift-ca\") pod \"ea0c2df3-91dd-40ae-8c3b-5588b8dc3517\" (UID: \"ea0c2df3-91dd-40ae-8c3b-5588b8dc3517\") "
Oct 30 17:36:55.338554 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.338534    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea0c2df3-91dd-40ae-8c3b-5588b8dc3517-maas-api-openshift-ca" pod "ea0c2df3-91dd-40ae-8c3b-5588b8dc3517" (UID: "ea0c2df3-91dd-40ae-8c3b-5588b8dc3517"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:36:55.338786 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.338773    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea0c2df3-91dd-40ae-8c3b-5588b8dc3517-maas-api-openshift-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:55.340265 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.340241    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea0c2df3-91dd-40ae-8c3b-5588b8dc3517-maas-api-xks-ca" pod "ea0c2df3-91dd-40ae-8c3b-5588b8dc3517" (UID: "ea0c2df3-91dd-40ae-8c3b-5588b8dc3517"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:36:55.340394 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.340344    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea0c2df3-91dd-40ae-8c3b-5588b8dc3517-kube-api-access-kmssk" pod "ea0c2df3-91dd-40ae-8c3b-5588b8dc3517" (UID: "ea0c2df3-91dd-40ae-8c3b-5588b8dc3517"). InnerVolumeSpecName "kube-api-access-kmssk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:36:55.440047 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.440004    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmssk\" (UniqueName: \"kubernetes.io/projected/ea0c2df3-91dd-40ae-8c3b-5588b8dc3517-kube-api-access-kmssk\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:55.440047 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.440041    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea0c2df3-91dd-40ae-8c3b-5588b8dc3517-maas-api-xks-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:55.517650 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.517608    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" containerID="cri-o://d123c293645970f64b703086d659a52d865fcdb1f3053d14bc8d3e36f5ea94ee" gracePeriod=45
Oct 30 17:36:55.670819 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.670777    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": dial tcp 10.133.0.38:8000: connect: connection refused"
Oct 30 17:36:55.714148 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.714115    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69083163-cvdt5"]
Oct 30 17:36:55.714736 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.714702    4433 cpu_manager.go:414] "containerMap: removing container" podUID="ea0c2df3-91dd-40ae-8c3b-5588b8dc3517" containerName="revoke-keys"
Oct 30 17:36:55.714736 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.714725    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea0c2df3-91dd-40ae-8c3b-5588b8dc3517" containerName="revoke-keys"
Oct 30 17:36:55.714917 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.714894    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea0c2df3-91dd-40ae-8c3b-5588b8dc3517" containerName="revoke-keys"
Oct 30 17:36:55.720372 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.719226    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69083163-cvdt5"
Oct 30 17:36:55.735023 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.734313    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69083163-cvdt5"]
Oct 30 17:36:55.842149 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.842119    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 17:36:55.845432 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.845388    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1d18482-74d1-4fe0-ae9c-08d49509ae7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69083163-cvdt5\" (UID: \"e1d18482-74d1-4fe0-ae9c-08d49509ae7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69083163-cvdt5"
Oct 30 17:36:55.845584 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.845462    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1d18482-74d1-4fe0-ae9c-08d49509ae7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69083163-cvdt5\" (UID: \"e1d18482-74d1-4fe0-ae9c-08d49509ae7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69083163-cvdt5"
Oct 30 17:36:55.845651 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.845589    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4vrk\" (UniqueName: \"kubernetes.io/projected/e1d18482-74d1-4fe0-ae9c-08d49509ae7b-kube-api-access-c4vrk\") pod \"maas-api-revoke-sub-69083163-cvdt5\" (UID: \"e1d18482-74d1-4fe0-ae9c-08d49509ae7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69083163-cvdt5"
Oct 30 17:36:55.947080 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.947034    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6409a840-87ab-4b90-9955-a2a05c47b50f-tls-certs\" (UniqueName: \"kubernetes.io/secret/6409a840-87ab-4b90-9955-a2a05c47b50f-tls-certs\") pod \"6409a840-87ab-4b90-9955-a2a05c47b50f\" (UID: \"6409a840-87ab-4b90-9955-a2a05c47b50f\") "
Oct 30 17:36:55.947242 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.947174    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6409a840-87ab-4b90-9955-a2a05c47b50f-kube-api-access-fzhmc\" (UniqueName: \"kubernetes.io/projected/6409a840-87ab-4b90-9955-a2a05c47b50f-kube-api-access-fzhmc\") pod \"6409a840-87ab-4b90-9955-a2a05c47b50f\" (UID: \"6409a840-87ab-4b90-9955-a2a05c47b50f\") "
Oct 30 17:36:55.947242 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.947211    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6409a840-87ab-4b90-9955-a2a05c47b50f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6409a840-87ab-4b90-9955-a2a05c47b50f-model-cache\") pod \"6409a840-87ab-4b90-9955-a2a05c47b50f\" (UID: \"6409a840-87ab-4b90-9955-a2a05c47b50f\") "
Oct 30 17:36:55.947380 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.947355    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6409a840-87ab-4b90-9955-a2a05c47b50f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6409a840-87ab-4b90-9955-a2a05c47b50f-tmp-dir\") pod \"6409a840-87ab-4b90-9955-a2a05c47b50f\" (UID: \"6409a840-87ab-4b90-9955-a2a05c47b50f\") "
Oct 30 17:36:55.947494 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.947471    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6409a840-87ab-4b90-9955-a2a05c47b50f-home\" (UniqueName: \"kubernetes.io/empty-dir/6409a840-87ab-4b90-9955-a2a05c47b50f-home\") pod \"6409a840-87ab-4b90-9955-a2a05c47b50f\" (UID: \"6409a840-87ab-4b90-9955-a2a05c47b50f\") "
Oct 30 17:36:55.947563 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.947538    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6409a840-87ab-4b90-9955-a2a05c47b50f-model-cache" pod "6409a840-87ab-4b90-9955-a2a05c47b50f" (UID: "6409a840-87ab-4b90-9955-a2a05c47b50f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:55.947626 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.947610    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6409a840-87ab-4b90-9955-a2a05c47b50f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6409a840-87ab-4b90-9955-a2a05c47b50f-dshm\") pod \"6409a840-87ab-4b90-9955-a2a05c47b50f\" (UID: \"6409a840-87ab-4b90-9955-a2a05c47b50f\") "
Oct 30 17:36:55.947861 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.947694    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6409a840-87ab-4b90-9955-a2a05c47b50f-tmp-dir" pod "6409a840-87ab-4b90-9955-a2a05c47b50f" (UID: "6409a840-87ab-4b90-9955-a2a05c47b50f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:55.947861 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.947768    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6409a840-87ab-4b90-9955-a2a05c47b50f-home" pod "6409a840-87ab-4b90-9955-a2a05c47b50f" (UID: "6409a840-87ab-4b90-9955-a2a05c47b50f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:55.948037 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.947949    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1d18482-74d1-4fe0-ae9c-08d49509ae7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69083163-cvdt5\" (UID: \"e1d18482-74d1-4fe0-ae9c-08d49509ae7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69083163-cvdt5"
Oct 30 17:36:55.948037 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.947995    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1d18482-74d1-4fe0-ae9c-08d49509ae7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69083163-cvdt5\" (UID: \"e1d18482-74d1-4fe0-ae9c-08d49509ae7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69083163-cvdt5"
Oct 30 17:36:55.948156 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.948117    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4vrk\" (UniqueName: \"kubernetes.io/projected/e1d18482-74d1-4fe0-ae9c-08d49509ae7b-kube-api-access-c4vrk\") pod \"maas-api-revoke-sub-69083163-cvdt5\" (UID: \"e1d18482-74d1-4fe0-ae9c-08d49509ae7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69083163-cvdt5"
Oct 30 17:36:55.948305 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.948283    4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6409a840-87ab-4b90-9955-a2a05c47b50f-tmp-dir\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:55.948381 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.948309    4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6409a840-87ab-4b90-9955-a2a05c47b50f-home\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:55.948381 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.948323    4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6409a840-87ab-4b90-9955-a2a05c47b50f-model-cache\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:55.948883 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.948815    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1d18482-74d1-4fe0-ae9c-08d49509ae7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69083163-cvdt5\" (UID: \"e1d18482-74d1-4fe0-ae9c-08d49509ae7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69083163-cvdt5"
Oct 30 17:36:55.950311 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.950285    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6409a840-87ab-4b90-9955-a2a05c47b50f-tls-certs" pod "6409a840-87ab-4b90-9955-a2a05c47b50f" (UID: "6409a840-87ab-4b90-9955-a2a05c47b50f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:36:55.950476 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.950397    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6409a840-87ab-4b90-9955-a2a05c47b50f-dshm" pod "6409a840-87ab-4b90-9955-a2a05c47b50f" (UID: "6409a840-87ab-4b90-9955-a2a05c47b50f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:36:55.950579 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.950531    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6409a840-87ab-4b90-9955-a2a05c47b50f-kube-api-access-fzhmc" pod "6409a840-87ab-4b90-9955-a2a05c47b50f" (UID: "6409a840-87ab-4b90-9955-a2a05c47b50f"). InnerVolumeSpecName "kube-api-access-fzhmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:36:55.951811 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.951788    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1d18482-74d1-4fe0-ae9c-08d49509ae7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69083163-cvdt5\" (UID: \"e1d18482-74d1-4fe0-ae9c-08d49509ae7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69083163-cvdt5"
Oct 30 17:36:55.972017 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:55.971975    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4vrk\" (UniqueName: \"kubernetes.io/projected/e1d18482-74d1-4fe0-ae9c-08d49509ae7b-kube-api-access-c4vrk\") pod \"maas-api-revoke-sub-69083163-cvdt5\" (UID: \"e1d18482-74d1-4fe0-ae9c-08d49509ae7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69083163-cvdt5"
Oct 30 17:36:56.028324 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:56.028283    4433 generic.go:356] "Generic (PLEG): container finished" podID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerID="d123c293645970f64b703086d659a52d865fcdb1f3053d14bc8d3e36f5ea94ee" exitCode=0
Oct 30 17:36:56.028324 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:56.028330    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"
Oct 30 17:36:56.028678 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:56.028366    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" event={"ID":"6409a840-87ab-4b90-9955-a2a05c47b50f","Type":"ContainerDied","Data":"d123c293645970f64b703086d659a52d865fcdb1f3053d14bc8d3e36f5ea94ee"}
Oct 30 17:36:56.028678 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:56.028439    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8" event={"ID":"6409a840-87ab-4b90-9955-a2a05c47b50f","Type":"ContainerDied","Data":"4611bdec303155726d2741133d7b5b73b6535955782e7052622a3b87fc9d2bfc"}
Oct 30 17:36:56.028678 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:56.028465    4433 scope.go:122] "RemoveContainer" containerID="d123c293645970f64b703086d659a52d865fcdb1f3053d14bc8d3e36f5ea94ee"
Oct 30 17:36:56.030842 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:56.030818    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef70ed1-qlxbp" event={"ID":"ea0c2df3-91dd-40ae-8c3b-5588b8dc3517","Type":"ContainerDied","Data":"ba0f73f8e553b18336052caa41932ce18e5468430058e1082077daf304f9ddc0"}
Oct 30 17:36:56.030975 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:56.030865    4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba0f73f8e553b18336052caa41932ce18e5468430058e1082077daf304f9ddc0"
Oct 30 17:36:56.030975 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:56.030874    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef70ed1-qlxbp"
Oct 30 17:36:56.040707 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:56.040685    4433 scope.go:122] "RemoveContainer" containerID="d123c293645970f64b703086d659a52d865fcdb1f3053d14bc8d3e36f5ea94ee"
Oct 30 17:36:56.041031 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:36:56.041011    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d123c293645970f64b703086d659a52d865fcdb1f3053d14bc8d3e36f5ea94ee\": container with ID starting with d123c293645970f64b703086d659a52d865fcdb1f3053d14bc8d3e36f5ea94ee not found: ID does not exist" containerID="d123c293645970f64b703086d659a52d865fcdb1f3053d14bc8d3e36f5ea94ee"
Oct 30 17:36:56.041103 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:56.041040    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d123c293645970f64b703086d659a52d865fcdb1f3053d14bc8d3e36f5ea94ee"} err="failed to get container status \"d123c293645970f64b703086d659a52d865fcdb1f3053d14bc8d3e36f5ea94ee\": rpc error: code = NotFound desc = could not find container \"d123c293645970f64b703086d659a52d865fcdb1f3053d14bc8d3e36f5ea94ee\": container with ID starting with d123c293645970f64b703086d659a52d865fcdb1f3053d14bc8d3e36f5ea94ee not found: ID does not exist"
Oct 30 17:36:56.045125 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:56.045102    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69083163-cvdt5"
Oct 30 17:36:56.049149 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:56.049123    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzhmc\" (UniqueName: \"kubernetes.io/projected/6409a840-87ab-4b90-9955-a2a05c47b50f-kube-api-access-fzhmc\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:56.049251 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:56.049155    4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6409a840-87ab-4b90-9955-a2a05c47b50f-dshm\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:56.049251 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:56.049169    4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6409a840-87ab-4b90-9955-a2a05c47b50f-tls-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:56.083189 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:56.083155    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"]
Oct 30 17:36:56.089173 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:56.089141    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b80d83/simulator-w3-b80d83-kserve-694b784ccf-dwdq8"]
Oct 30 17:36:56.353440 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:56.353204    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69083163-cvdt5"]
Oct 30 17:36:56.356688 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:36:56.356658    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1d18482_74d1_4fe0_ae9c_08d49509ae7b.slice/crio-e6191dd417f69107f558f0a2b32f84ae016136b87e8fd9c8a3fd3c415df91735 WatchSource:0}: Error finding container e6191dd417f69107f558f0a2b32f84ae016136b87e8fd9c8a3fd3c415df91735: Status 404 returned error can't find the container with id e6191dd417f69107f558f0a2b32f84ae016136b87e8fd9c8a3fd3c415df91735
Oct 30 17:36:56.519814 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:56.519756    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"]
Oct 30 17:36:56.520473 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:56.520437    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s" podUID="98a7e2bc-f103-44c3-b5a4-4f93647a28e7" containerName="istio-proxy" containerID="cri-o://21c04ed531636f8a1bcdc72e1f891223906b38224c65a1bdb8e8bf4325423aa9" gracePeriod=30
Oct 30 17:36:56.977121 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:36:56.977033    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:36:57.038165 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:57.038128    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69083163-cvdt5" event={"ID":"e1d18482-74d1-4fe0-ae9c-08d49509ae7b","Type":"ContainerStarted","Data":"4247a06bfbc0a6a77b33f186566c80d7a1be6385bcaf48e1c59c588adc5921de"}
Oct 30 17:36:57.038370 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:57.038192    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69083163-cvdt5" event={"ID":"e1d18482-74d1-4fe0-ae9c-08d49509ae7b","Type":"ContainerStarted","Data":"e6191dd417f69107f558f0a2b32f84ae016136b87e8fd9c8a3fd3c415df91735"}
Oct 30 17:36:57.077719 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:57.077665    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69083163-cvdt5" podStartSLOduration=2.077640064 podStartE2EDuration="2.077640064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:36:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:57.076842032 +0000 UTC m=+2993.511744251" watchObservedRunningTime="2026-10-30 17:36:57.077640064 +0000 UTC m=+2993.512542271"
Oct 30 17:36:57.701399 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:57.701364    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" path="/var/lib/kubelet/pods/6409a840-87ab-4b90-9955-a2a05c47b50f/volumes"
Oct 30 17:36:58.052275 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:58.052174    4433 generic.go:356] "Generic (PLEG): container finished" podID="e1d18482-74d1-4fe0-ae9c-08d49509ae7b" containerID="4247a06bfbc0a6a77b33f186566c80d7a1be6385bcaf48e1c59c588adc5921de" exitCode=0
Oct 30 17:36:58.052275 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:58.052236    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69083163-cvdt5" event={"ID":"e1d18482-74d1-4fe0-ae9c-08d49509ae7b","Type":"ContainerDied","Data":"4247a06bfbc0a6a77b33f186566c80d7a1be6385bcaf48e1c59c588adc5921de"}
Oct 30 17:36:59.200378 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:59.200353    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69083163-cvdt5"
Oct 30 17:36:59.325953 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:59.325866    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1d18482-74d1-4fe0-ae9c-08d49509ae7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1d18482-74d1-4fe0-ae9c-08d49509ae7b-maas-api-openshift-ca\") pod \"e1d18482-74d1-4fe0-ae9c-08d49509ae7b\" (UID: \"e1d18482-74d1-4fe0-ae9c-08d49509ae7b\") "
Oct 30 17:36:59.326104 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:59.326021    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1d18482-74d1-4fe0-ae9c-08d49509ae7b-kube-api-access-c4vrk\" (UniqueName: \"kubernetes.io/projected/e1d18482-74d1-4fe0-ae9c-08d49509ae7b-kube-api-access-c4vrk\") pod \"e1d18482-74d1-4fe0-ae9c-08d49509ae7b\" (UID: \"e1d18482-74d1-4fe0-ae9c-08d49509ae7b\") "
Oct 30 17:36:59.326104 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:59.326064    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1d18482-74d1-4fe0-ae9c-08d49509ae7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1d18482-74d1-4fe0-ae9c-08d49509ae7b-maas-api-xks-ca\") pod \"e1d18482-74d1-4fe0-ae9c-08d49509ae7b\" (UID: \"e1d18482-74d1-4fe0-ae9c-08d49509ae7b\") "
Oct 30 17:36:59.326288 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:59.326261    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1d18482-74d1-4fe0-ae9c-08d49509ae7b-maas-api-openshift-ca" pod "e1d18482-74d1-4fe0-ae9c-08d49509ae7b" (UID: "e1d18482-74d1-4fe0-ae9c-08d49509ae7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:36:59.326443 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:59.326426    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1d18482-74d1-4fe0-ae9c-08d49509ae7b-maas-api-openshift-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:59.328106 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:59.328080    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1d18482-74d1-4fe0-ae9c-08d49509ae7b-kube-api-access-c4vrk" pod "e1d18482-74d1-4fe0-ae9c-08d49509ae7b" (UID: "e1d18482-74d1-4fe0-ae9c-08d49509ae7b"). InnerVolumeSpecName "kube-api-access-c4vrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:36:59.328212 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:59.328192    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1d18482-74d1-4fe0-ae9c-08d49509ae7b-maas-api-xks-ca" pod "e1d18482-74d1-4fe0-ae9c-08d49509ae7b" (UID: "e1d18482-74d1-4fe0-ae9c-08d49509ae7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:36:59.427711 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:59.427679    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4vrk\" (UniqueName: \"kubernetes.io/projected/e1d18482-74d1-4fe0-ae9c-08d49509ae7b-kube-api-access-c4vrk\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:36:59.427711 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:36:59.427709    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1d18482-74d1-4fe0-ae9c-08d49509ae7b-maas-api-xks-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:00.066892 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:00.066806    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69083163-cvdt5"
Oct 30 17:37:00.066892 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:00.066836    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69083163-cvdt5" event={"ID":"e1d18482-74d1-4fe0-ae9c-08d49509ae7b","Type":"ContainerDied","Data":"e6191dd417f69107f558f0a2b32f84ae016136b87e8fd9c8a3fd3c415df91735"}
Oct 30 17:37:00.066892 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:00.066883    4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6191dd417f69107f558f0a2b32f84ae016136b87e8fd9c8a3fd3c415df91735"
Oct 30 17:37:01.827329 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:01.827297    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"
Oct 30 17:37:01.959025 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:01.958975    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-credential-socket\") pod \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") "
Oct 30 17:37:01.959210 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:01.959049    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-istio-token\" (UniqueName: \"kubernetes.io/projected/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-istio-token\") pod \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") "
Oct 30 17:37:01.959210 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:01.959081    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-kube-api-access-kq6p7\" (UniqueName: \"kubernetes.io/projected/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-kube-api-access-kq6p7\") pod \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") "
Oct 30 17:37:01.959210 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:01.959180    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-istio-envoy\") pod \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") "
Oct 30 17:37:01.959450 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:01.959230    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-workload-socket\") pod \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") "
Oct 30 17:37:01.959450 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:01.959317    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-istio-data\") pod \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") "
Oct 30 17:37:01.959450 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:01.959390    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-istiod-ca-cert\") pod \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") "
Oct 30 17:37:01.959629 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:01.959484    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-workload-certs\") pod \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") "
Oct 30 17:37:01.959629 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:01.959596    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-istio-podinfo\") pod \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\" (UID: \"98a7e2bc-f103-44c3-b5a4-4f93647a28e7\") "
Oct 30 17:37:01.959752 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:01.959707    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-istio-data" pod "98a7e2bc-f103-44c3-b5a4-4f93647a28e7" (UID: "98a7e2bc-f103-44c3-b5a4-4f93647a28e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:37:01.960806 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:01.959931    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-workload-socket" pod "98a7e2bc-f103-44c3-b5a4-4f93647a28e7" (UID: "98a7e2bc-f103-44c3-b5a4-4f93647a28e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:37:01.960806 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:01.960125    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:01.960806 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:01.960145    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:01.960806 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:01.960475    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-istiod-ca-cert" pod "98a7e2bc-f103-44c3-b5a4-4f93647a28e7" (UID: "98a7e2bc-f103-44c3-b5a4-4f93647a28e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:37:01.960806 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:01.960775    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-workload-certs" pod "98a7e2bc-f103-44c3-b5a4-4f93647a28e7" (UID: "98a7e2bc-f103-44c3-b5a4-4f93647a28e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:37:01.961345 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:01.961301    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-credential-socket" pod "98a7e2bc-f103-44c3-b5a4-4f93647a28e7" (UID: "98a7e2bc-f103-44c3-b5a4-4f93647a28e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:37:01.962131 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:01.962104    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-kube-api-access-kq6p7" pod "98a7e2bc-f103-44c3-b5a4-4f93647a28e7" (UID: "98a7e2bc-f103-44c3-b5a4-4f93647a28e7"). InnerVolumeSpecName "kube-api-access-kq6p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:37:01.962274 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:01.962143    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-istio-token" pod "98a7e2bc-f103-44c3-b5a4-4f93647a28e7" (UID: "98a7e2bc-f103-44c3-b5a4-4f93647a28e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:37:01.962693 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:01.962669    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-istio-envoy" pod "98a7e2bc-f103-44c3-b5a4-4f93647a28e7" (UID: "98a7e2bc-f103-44c3-b5a4-4f93647a28e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:37:01.962898 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:01.962870    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-istio-podinfo" pod "98a7e2bc-f103-44c3-b5a4-4f93647a28e7" (UID: "98a7e2bc-f103-44c3-b5a4-4f93647a28e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:37:02.039389 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:02.039347    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"]
Oct 30 17:37:02.049488 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:02.049391    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"]
Oct 30 17:37:02.061153 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:02.061115    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:02.061153 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:02.061148    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:02.061368 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:02.061162    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq6p7\" (UniqueName: \"kubernetes.io/projected/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-kube-api-access-kq6p7\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:02.061368 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:02.061177    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:02.061368 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:02.061189    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:02.061368 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:02.061200    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:02.061368 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:02.061211    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98a7e2bc-f103-44c3-b5a4-4f93647a28e7-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:02.081519 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:02.081487    4433 generic.go:356] "Generic (PLEG): container finished" podID="98a7e2bc-f103-44c3-b5a4-4f93647a28e7" containerID="21c04ed531636f8a1bcdc72e1f891223906b38224c65a1bdb8e8bf4325423aa9" exitCode=0
Oct 30 17:37:02.081674 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:02.081543    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"
Oct 30 17:37:02.081674 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:02.081568    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s" event={"ID":"98a7e2bc-f103-44c3-b5a4-4f93647a28e7","Type":"ContainerDied","Data":"21c04ed531636f8a1bcdc72e1f891223906b38224c65a1bdb8e8bf4325423aa9"}
Oct 30 17:37:02.081674 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:02.081634    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s" event={"ID":"98a7e2bc-f103-44c3-b5a4-4f93647a28e7","Type":"ContainerDied","Data":"57abc099ccb4906eee0df7369d8138b44281157313ffa5ab8eb743072df4650d"}
Oct 30 17:37:02.081674 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:02.081666    4433 scope.go:122] "RemoveContainer" containerID="21c04ed531636f8a1bcdc72e1f891223906b38224c65a1bdb8e8bf4325423aa9"
Oct 30 17:37:02.094732 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:02.094706    4433 scope.go:122] "RemoveContainer" containerID="21c04ed531636f8a1bcdc72e1f891223906b38224c65a1bdb8e8bf4325423aa9"
Oct 30 17:37:02.095077 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:37:02.095054    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21c04ed531636f8a1bcdc72e1f891223906b38224c65a1bdb8e8bf4325423aa9\": container with ID starting with 21c04ed531636f8a1bcdc72e1f891223906b38224c65a1bdb8e8bf4325423aa9 not found: ID does not exist" containerID="21c04ed531636f8a1bcdc72e1f891223906b38224c65a1bdb8e8bf4325423aa9"
Oct 30 17:37:02.095154 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:02.095087    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21c04ed531636f8a1bcdc72e1f891223906b38224c65a1bdb8e8bf4325423aa9"} err="failed to get container status \"21c04ed531636f8a1bcdc72e1f891223906b38224c65a1bdb8e8bf4325423aa9\": rpc error: code = NotFound desc = could not find container \"21c04ed531636f8a1bcdc72e1f891223906b38224c65a1bdb8e8bf4325423aa9\": container with ID starting with 21c04ed531636f8a1bcdc72e1f891223906b38224c65a1bdb8e8bf4325423aa9 not found: ID does not exist"
Oct 30 17:37:02.181226 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:02.181175    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"]
Oct 30 17:37:02.227821 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:02.227784    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-82243b-openshift-default-6b684969d5-6vd6s"]
Oct 30 17:37:03.717829 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:03.713217    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98a7e2bc-f103-44c3-b5a4-4f93647a28e7" path="/var/lib/kubelet/pods/98a7e2bc-f103-44c3-b5a4-4f93647a28e7/volumes"
Oct 30 17:37:07.384853 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:07.367284    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"]
Oct 30 17:37:07.384853 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:07.367554    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7" podUID="ea8fea41-d6c6-414b-8a0d-770c435d514c" containerName="istio-proxy" containerID="cri-o://7e72b79430fac5e029ffbc3bcf7602b8ebc2b0d7af89cfb018279301a854497d" gracePeriod=30
Oct 30 17:37:12.555825 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:12.555798    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"
Oct 30 17:37:12.635729 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:12.635631    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea8fea41-d6c6-414b-8a0d-770c435d514c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea8fea41-d6c6-414b-8a0d-770c435d514c-istio-podinfo\") pod \"ea8fea41-d6c6-414b-8a0d-770c435d514c\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") "
Oct 30 17:37:12.635729 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:12.635677    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea8fea41-d6c6-414b-8a0d-770c435d514c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea8fea41-d6c6-414b-8a0d-770c435d514c-istiod-ca-cert\") pod \"ea8fea41-d6c6-414b-8a0d-770c435d514c\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") "
Oct 30 17:37:12.635729 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:12.635729    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea8fea41-d6c6-414b-8a0d-770c435d514c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea8fea41-d6c6-414b-8a0d-770c435d514c-credential-socket\") pod \"ea8fea41-d6c6-414b-8a0d-770c435d514c\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") "
Oct 30 17:37:12.636048 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:12.635760    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea8fea41-d6c6-414b-8a0d-770c435d514c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea8fea41-d6c6-414b-8a0d-770c435d514c-istio-envoy\") pod \"ea8fea41-d6c6-414b-8a0d-770c435d514c\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") "
Oct 30 17:37:12.636048 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:12.635795    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea8fea41-d6c6-414b-8a0d-770c435d514c-istio-token\" (UniqueName: \"kubernetes.io/projected/ea8fea41-d6c6-414b-8a0d-770c435d514c-istio-token\") pod \"ea8fea41-d6c6-414b-8a0d-770c435d514c\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") "
Oct 30 17:37:12.636048 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:12.635865    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea8fea41-d6c6-414b-8a0d-770c435d514c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea8fea41-d6c6-414b-8a0d-770c435d514c-workload-certs\") pod \"ea8fea41-d6c6-414b-8a0d-770c435d514c\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") "
Oct 30 17:37:12.636048 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:12.636022    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea8fea41-d6c6-414b-8a0d-770c435d514c-kube-api-access-7svfc\" (UniqueName: \"kubernetes.io/projected/ea8fea41-d6c6-414b-8a0d-770c435d514c-kube-api-access-7svfc\") pod \"ea8fea41-d6c6-414b-8a0d-770c435d514c\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") "
Oct 30 17:37:12.636275 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:12.636143    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea8fea41-d6c6-414b-8a0d-770c435d514c-credential-socket" pod "ea8fea41-d6c6-414b-8a0d-770c435d514c" (UID: "ea8fea41-d6c6-414b-8a0d-770c435d514c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:37:12.636508 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:12.636456    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea8fea41-d6c6-414b-8a0d-770c435d514c-istiod-ca-cert" pod "ea8fea41-d6c6-414b-8a0d-770c435d514c" (UID: "ea8fea41-d6c6-414b-8a0d-770c435d514c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:37:12.636508 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:12.636450    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea8fea41-d6c6-414b-8a0d-770c435d514c-workload-certs" pod "ea8fea41-d6c6-414b-8a0d-770c435d514c" (UID: "ea8fea41-d6c6-414b-8a0d-770c435d514c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:37:12.636836 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:12.636504    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea8fea41-d6c6-414b-8a0d-770c435d514c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea8fea41-d6c6-414b-8a0d-770c435d514c-istio-data\") pod \"ea8fea41-d6c6-414b-8a0d-770c435d514c\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") "
Oct 30 17:37:12.636836 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:12.636549    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea8fea41-d6c6-414b-8a0d-770c435d514c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea8fea41-d6c6-414b-8a0d-770c435d514c-workload-socket\") pod \"ea8fea41-d6c6-414b-8a0d-770c435d514c\" (UID: \"ea8fea41-d6c6-414b-8a0d-770c435d514c\") "
Oct 30 17:37:12.636836 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:12.636791    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea8fea41-d6c6-414b-8a0d-770c435d514c-istio-data" pod "ea8fea41-d6c6-414b-8a0d-770c435d514c" (UID: "ea8fea41-d6c6-414b-8a0d-770c435d514c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:37:12.637045 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:12.637003    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea8fea41-d6c6-414b-8a0d-770c435d514c-workload-socket" pod "ea8fea41-d6c6-414b-8a0d-770c435d514c" (UID: "ea8fea41-d6c6-414b-8a0d-770c435d514c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:37:12.637141 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:12.637116    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea8fea41-d6c6-414b-8a0d-770c435d514c-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:12.637141 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:12.637141    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea8fea41-d6c6-414b-8a0d-770c435d514c-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:12.637333 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:12.637157    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea8fea41-d6c6-414b-8a0d-770c435d514c-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:12.637333 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:12.637171    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea8fea41-d6c6-414b-8a0d-770c435d514c-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:12.637333 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:12.637184    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea8fea41-d6c6-414b-8a0d-770c435d514c-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:12.638543 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:12.638506    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea8fea41-d6c6-414b-8a0d-770c435d514c-istio-token" pod "ea8fea41-d6c6-414b-8a0d-770c435d514c" (UID: "ea8fea41-d6c6-414b-8a0d-770c435d514c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:37:12.638543 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:12.638523    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea8fea41-d6c6-414b-8a0d-770c435d514c-istio-podinfo" pod "ea8fea41-d6c6-414b-8a0d-770c435d514c" (UID: "ea8fea41-d6c6-414b-8a0d-770c435d514c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:37:12.638728 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:12.638672    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea8fea41-d6c6-414b-8a0d-770c435d514c-istio-envoy" pod "ea8fea41-d6c6-414b-8a0d-770c435d514c" (UID: "ea8fea41-d6c6-414b-8a0d-770c435d514c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:37:12.639049 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:12.639025    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea8fea41-d6c6-414b-8a0d-770c435d514c-kube-api-access-7svfc" pod "ea8fea41-d6c6-414b-8a0d-770c435d514c" (UID: "ea8fea41-d6c6-414b-8a0d-770c435d514c"). InnerVolumeSpecName "kube-api-access-7svfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:37:12.738689 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:12.738646    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7svfc\" (UniqueName: \"kubernetes.io/projected/ea8fea41-d6c6-414b-8a0d-770c435d514c-kube-api-access-7svfc\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:12.738689 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:12.738680    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea8fea41-d6c6-414b-8a0d-770c435d514c-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:12.738689 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:12.738697    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea8fea41-d6c6-414b-8a0d-770c435d514c-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:12.738936 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:12.738707    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea8fea41-d6c6-414b-8a0d-770c435d514c-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:13.162275 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:13.162236    4433 generic.go:356] "Generic (PLEG): container finished" podID="ea8fea41-d6c6-414b-8a0d-770c435d514c" containerID="7e72b79430fac5e029ffbc3bcf7602b8ebc2b0d7af89cfb018279301a854497d" exitCode=0
Oct 30 17:37:13.162487 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:13.162285    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"
Oct 30 17:37:13.162487 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:13.162301    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7" event={"ID":"ea8fea41-d6c6-414b-8a0d-770c435d514c","Type":"ContainerDied","Data":"7e72b79430fac5e029ffbc3bcf7602b8ebc2b0d7af89cfb018279301a854497d"}
Oct 30 17:37:13.162487 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:13.162351    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7" event={"ID":"ea8fea41-d6c6-414b-8a0d-770c435d514c","Type":"ContainerDied","Data":"6bb9d2548a58e8dc5f3fe94ddb1a50b40458afe5558d7770bce608721ed77953"}
Oct 30 17:37:13.162487 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:13.162381    4433 scope.go:122] "RemoveContainer" containerID="7e72b79430fac5e029ffbc3bcf7602b8ebc2b0d7af89cfb018279301a854497d"
Oct 30 17:37:13.172740 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:13.172719    4433 scope.go:122] "RemoveContainer" containerID="7e72b79430fac5e029ffbc3bcf7602b8ebc2b0d7af89cfb018279301a854497d"
Oct 30 17:37:13.173034 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:37:13.173014    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e72b79430fac5e029ffbc3bcf7602b8ebc2b0d7af89cfb018279301a854497d\": container with ID starting with 7e72b79430fac5e029ffbc3bcf7602b8ebc2b0d7af89cfb018279301a854497d not found: ID does not exist" containerID="7e72b79430fac5e029ffbc3bcf7602b8ebc2b0d7af89cfb018279301a854497d"
Oct 30 17:37:13.173084 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:13.173045    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e72b79430fac5e029ffbc3bcf7602b8ebc2b0d7af89cfb018279301a854497d"} err="failed to get container status \"7e72b79430fac5e029ffbc3bcf7602b8ebc2b0d7af89cfb018279301a854497d\": rpc error: code = NotFound desc = could not find container \"7e72b79430fac5e029ffbc3bcf7602b8ebc2b0d7af89cfb018279301a854497d\": container with ID starting with 7e72b79430fac5e029ffbc3bcf7602b8ebc2b0d7af89cfb018279301a854497d not found: ID does not exist"
Oct 30 17:37:13.213972 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:13.213932    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"]
Oct 30 17:37:13.225996 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:13.225960    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-tzqm7"]
Oct 30 17:37:13.695870 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:13.695829    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea8fea41-d6c6-414b-8a0d-770c435d514c" path="/var/lib/kubelet/pods/ea8fea41-d6c6-414b-8a0d-770c435d514c/volumes"
Oct 30 17:37:16.648484 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:16.647708    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:37:17.062113 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.062070    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main" containerID="cri-o://f48a752853a7011ac6c56e390b455a70676c284ca85078d7cb6d29172b4af3c7" gracePeriod=45
Oct 30 17:37:17.070254 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.070208    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" containerID="cri-o://dc996a98dac4fb8db5075b13e2ff966a2f239ac37a4ee1bd63e9d94cf3d76993" gracePeriod=45
Oct 30 17:37:17.205574 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.189853    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": dial tcp 10.133.0.39:8000: connect: connection refused"
Oct 30 17:37:17.231751 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.231716    4433 generic.go:356] "Generic (PLEG): container finished" podID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerID="f48a752853a7011ac6c56e390b455a70676c284ca85078d7cb6d29172b4af3c7" exitCode=0
Oct 30 17:37:17.231925 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.231906    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" event={"ID":"2e1669d0-fffa-471f-9275-de0fa70d98e7","Type":"ContainerDied","Data":"f48a752853a7011ac6c56e390b455a70676c284ca85078d7cb6d29172b4af3c7"}
Oct 30 17:37:17.554181 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.554136    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 17:37:17.556924 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.556895    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:37:17.613126 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.613041    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e1669d0-fffa-471f-9275-de0fa70d98e7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e1669d0-fffa-471f-9275-de0fa70d98e7-dshm\") pod \"2e1669d0-fffa-471f-9275-de0fa70d98e7\" (UID: \"2e1669d0-fffa-471f-9275-de0fa70d98e7\") "
Oct 30 17:37:17.613126 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.613120    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab427c7b-928a-4efd-a355-6633379fc574-kube-api-access-h4rt2\" (UniqueName: \"kubernetes.io/projected/ab427c7b-928a-4efd-a355-6633379fc574-kube-api-access-h4rt2\") pod \"ab427c7b-928a-4efd-a355-6633379fc574\" (UID: \"ab427c7b-928a-4efd-a355-6633379fc574\") "
Oct 30 17:37:17.613367 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.613156    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e1669d0-fffa-471f-9275-de0fa70d98e7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e1669d0-fffa-471f-9275-de0fa70d98e7-model-cache\") pod \"2e1669d0-fffa-471f-9275-de0fa70d98e7\" (UID: \"2e1669d0-fffa-471f-9275-de0fa70d98e7\") "
Oct 30 17:37:17.613367 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.613185    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab427c7b-928a-4efd-a355-6633379fc574-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab427c7b-928a-4efd-a355-6633379fc574-dshm\") pod \"ab427c7b-928a-4efd-a355-6633379fc574\" (UID: \"ab427c7b-928a-4efd-a355-6633379fc574\") "
Oct 30 17:37:17.613367 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.613227    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e1669d0-fffa-471f-9275-de0fa70d98e7-home\" (UniqueName: \"kubernetes.io/empty-dir/2e1669d0-fffa-471f-9275-de0fa70d98e7-home\") pod \"2e1669d0-fffa-471f-9275-de0fa70d98e7\" (UID: \"2e1669d0-fffa-471f-9275-de0fa70d98e7\") "
Oct 30 17:37:17.613367 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.613270    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e1669d0-fffa-471f-9275-de0fa70d98e7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e1669d0-fffa-471f-9275-de0fa70d98e7-tmp-dir\") pod \"2e1669d0-fffa-471f-9275-de0fa70d98e7\" (UID: \"2e1669d0-fffa-471f-9275-de0fa70d98e7\") "
Oct 30 17:37:17.613367 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.613331    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab427c7b-928a-4efd-a355-6633379fc574-home\" (UniqueName: \"kubernetes.io/empty-dir/ab427c7b-928a-4efd-a355-6633379fc574-home\") pod \"ab427c7b-928a-4efd-a355-6633379fc574\" (UID: \"ab427c7b-928a-4efd-a355-6633379fc574\") "
Oct 30 17:37:17.613367 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.613364    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e1669d0-fffa-471f-9275-de0fa70d98e7-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e1669d0-fffa-471f-9275-de0fa70d98e7-tls-certs\") pod \"2e1669d0-fffa-471f-9275-de0fa70d98e7\" (UID: \"2e1669d0-fffa-471f-9275-de0fa70d98e7\") "
Oct 30 17:37:17.613722 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.613504    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab427c7b-928a-4efd-a355-6633379fc574-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab427c7b-928a-4efd-a355-6633379fc574-tls-certs\") pod \"ab427c7b-928a-4efd-a355-6633379fc574\" (UID: \"ab427c7b-928a-4efd-a355-6633379fc574\") "
Oct 30 17:37:17.613722 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.613547    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab427c7b-928a-4efd-a355-6633379fc574-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab427c7b-928a-4efd-a355-6633379fc574-tmp-dir\") pod \"ab427c7b-928a-4efd-a355-6633379fc574\" (UID: \"ab427c7b-928a-4efd-a355-6633379fc574\") "
Oct 30 17:37:17.613722 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.613596    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e1669d0-fffa-471f-9275-de0fa70d98e7-kube-api-access-vl4ss\" (UniqueName: \"kubernetes.io/projected/2e1669d0-fffa-471f-9275-de0fa70d98e7-kube-api-access-vl4ss\") pod \"2e1669d0-fffa-471f-9275-de0fa70d98e7\" (UID: \"2e1669d0-fffa-471f-9275-de0fa70d98e7\") "
Oct 30 17:37:17.613722 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.613642    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab427c7b-928a-4efd-a355-6633379fc574-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab427c7b-928a-4efd-a355-6633379fc574-model-cache\") pod \"ab427c7b-928a-4efd-a355-6633379fc574\" (UID: \"ab427c7b-928a-4efd-a355-6633379fc574\") "
Oct 30 17:37:17.614643 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.614329    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab427c7b-928a-4efd-a355-6633379fc574-model-cache" pod "ab427c7b-928a-4efd-a355-6633379fc574" (UID: "ab427c7b-928a-4efd-a355-6633379fc574"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:37:17.614774 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.614705    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e1669d0-fffa-471f-9275-de0fa70d98e7-tmp-dir" pod "2e1669d0-fffa-471f-9275-de0fa70d98e7" (UID: "2e1669d0-fffa-471f-9275-de0fa70d98e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:37:17.615563 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.615353    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab427c7b-928a-4efd-a355-6633379fc574-home" pod "ab427c7b-928a-4efd-a355-6633379fc574" (UID: "ab427c7b-928a-4efd-a355-6633379fc574"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:37:17.615788 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.615593    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab427c7b-928a-4efd-a355-6633379fc574-tmp-dir" pod "ab427c7b-928a-4efd-a355-6633379fc574" (UID: "ab427c7b-928a-4efd-a355-6633379fc574"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:37:17.616186 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.615895    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e1669d0-fffa-471f-9275-de0fa70d98e7-home" pod "2e1669d0-fffa-471f-9275-de0fa70d98e7" (UID: "2e1669d0-fffa-471f-9275-de0fa70d98e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:37:17.616186 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.616141    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e1669d0-fffa-471f-9275-de0fa70d98e7-model-cache" pod "2e1669d0-fffa-471f-9275-de0fa70d98e7" (UID: "2e1669d0-fffa-471f-9275-de0fa70d98e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:37:17.617121 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.617095    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e1669d0-fffa-471f-9275-de0fa70d98e7-dshm" pod "2e1669d0-fffa-471f-9275-de0fa70d98e7" (UID: "2e1669d0-fffa-471f-9275-de0fa70d98e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:37:17.626597 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.626557    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab427c7b-928a-4efd-a355-6633379fc574-dshm" pod "ab427c7b-928a-4efd-a355-6633379fc574" (UID: "ab427c7b-928a-4efd-a355-6633379fc574"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:37:17.627580 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.627539    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab427c7b-928a-4efd-a355-6633379fc574-tls-certs" pod "ab427c7b-928a-4efd-a355-6633379fc574" (UID: "ab427c7b-928a-4efd-a355-6633379fc574"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:37:17.627580 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.627541    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e1669d0-fffa-471f-9275-de0fa70d98e7-tls-certs" pod "2e1669d0-fffa-471f-9275-de0fa70d98e7" (UID: "2e1669d0-fffa-471f-9275-de0fa70d98e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:37:17.627743 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.627590    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e1669d0-fffa-471f-9275-de0fa70d98e7-kube-api-access-vl4ss" pod "2e1669d0-fffa-471f-9275-de0fa70d98e7" (UID: "2e1669d0-fffa-471f-9275-de0fa70d98e7"). InnerVolumeSpecName "kube-api-access-vl4ss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:37:17.628052 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.628020    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab427c7b-928a-4efd-a355-6633379fc574-kube-api-access-h4rt2" pod "ab427c7b-928a-4efd-a355-6633379fc574" (UID: "ab427c7b-928a-4efd-a355-6633379fc574"). InnerVolumeSpecName "kube-api-access-h4rt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:37:17.715266 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.715236    4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab427c7b-928a-4efd-a355-6633379fc574-tls-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:17.715266 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.715268    4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab427c7b-928a-4efd-a355-6633379fc574-tmp-dir\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:17.715668 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.715282    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl4ss\" (UniqueName: \"kubernetes.io/projected/2e1669d0-fffa-471f-9275-de0fa70d98e7-kube-api-access-vl4ss\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:17.715668 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.715296    4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab427c7b-928a-4efd-a355-6633379fc574-model-cache\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:17.715668 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.715310    4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e1669d0-fffa-471f-9275-de0fa70d98e7-dshm\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:17.715668 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.715323    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4rt2\" (UniqueName: \"kubernetes.io/projected/ab427c7b-928a-4efd-a355-6633379fc574-kube-api-access-h4rt2\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:17.715668 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.715339    4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e1669d0-fffa-471f-9275-de0fa70d98e7-model-cache\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:17.715668 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.715352    4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab427c7b-928a-4efd-a355-6633379fc574-dshm\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:17.715668 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.715364    4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e1669d0-fffa-471f-9275-de0fa70d98e7-home\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:17.715668 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.715376    4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e1669d0-fffa-471f-9275-de0fa70d98e7-tmp-dir\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:17.715668 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.715389    4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab427c7b-928a-4efd-a355-6633379fc574-home\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:17.715668 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:17.715416    4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e1669d0-fffa-471f-9275-de0fa70d98e7-tls-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:18.238593 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:18.238560    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"
Oct 30 17:37:18.238790 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:18.238593    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" event={"ID":"2e1669d0-fffa-471f-9275-de0fa70d98e7","Type":"ContainerDied","Data":"d520cc4d3fc9703dda3911c0221c00aaec7c0c75e77f4cf0bec0aa3a179b5a64"}
Oct 30 17:37:18.238790 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:18.238682    4433 scope.go:122] "RemoveContainer" containerID="f48a752853a7011ac6c56e390b455a70676c284ca85078d7cb6d29172b4af3c7"
Oct 30 17:37:18.240836 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:18.240779    4433 generic.go:356] "Generic (PLEG): container finished" podID="ab427c7b-928a-4efd-a355-6633379fc574" containerID="dc996a98dac4fb8db5075b13e2ff966a2f239ac37a4ee1bd63e9d94cf3d76993" exitCode=0
Oct 30 17:37:18.240962 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:18.240847    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" event={"ID":"ab427c7b-928a-4efd-a355-6633379fc574","Type":"ContainerDied","Data":"dc996a98dac4fb8db5075b13e2ff966a2f239ac37a4ee1bd63e9d94cf3d76993"}
Oct 30 17:37:18.240962 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:18.240877    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz" event={"ID":"ab427c7b-928a-4efd-a355-6633379fc574","Type":"ContainerDied","Data":"a5c5a54fc77156e33710dca18399df33b926376b8f13c7ca2a2fde08a3fd6d2c"}
Oct 30 17:37:18.241076 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:18.240978    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"
Oct 30 17:37:18.252554 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:18.252527    4433 scope.go:122] "RemoveContainer" containerID="dc996a98dac4fb8db5075b13e2ff966a2f239ac37a4ee1bd63e9d94cf3d76993"
Oct 30 17:37:18.262650 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:18.262631    4433 scope.go:122] "RemoveContainer" containerID="dc996a98dac4fb8db5075b13e2ff966a2f239ac37a4ee1bd63e9d94cf3d76993"
Oct 30 17:37:18.262979 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:37:18.262960    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc996a98dac4fb8db5075b13e2ff966a2f239ac37a4ee1bd63e9d94cf3d76993\": container with ID starting with dc996a98dac4fb8db5075b13e2ff966a2f239ac37a4ee1bd63e9d94cf3d76993 not found: ID does not exist" containerID="dc996a98dac4fb8db5075b13e2ff966a2f239ac37a4ee1bd63e9d94cf3d76993"
Oct 30 17:37:18.263026 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:18.262989    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc996a98dac4fb8db5075b13e2ff966a2f239ac37a4ee1bd63e9d94cf3d76993"} err="failed to get container status \"dc996a98dac4fb8db5075b13e2ff966a2f239ac37a4ee1bd63e9d94cf3d76993\": rpc error: code = NotFound desc = could not find container \"dc996a98dac4fb8db5075b13e2ff966a2f239ac37a4ee1bd63e9d94cf3d76993\": container with ID starting with dc996a98dac4fb8db5075b13e2ff966a2f239ac37a4ee1bd63e9d94cf3d76993 not found: ID does not exist"
Oct 30 17:37:18.289906 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:18.289872    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"]
Oct 30 17:37:18.299357 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:18.299327    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-86f50a/simulator-w5-86f50a-kserve-57c9654f4d-hp6pz"]
Oct 30 17:37:18.334063 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:18.333987    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"]
Oct 30 17:37:18.337992 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:18.337967    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6"]
Oct 30 17:37:18.484984 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:18.484938    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/premium-w5-86f50a-kserve-88687f89c-gpkw6" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:37:19.698855 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:19.698818    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" path="/var/lib/kubelet/pods/2e1669d0-fffa-471f-9275-de0fa70d98e7/volumes"
Oct 30 17:37:19.700078 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:19.699560    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab427c7b-928a-4efd-a355-6633379fc574" path="/var/lib/kubelet/pods/ab427c7b-928a-4efd-a355-6633379fc574/volumes"
Oct 30 17:37:26.978348 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:37:26.978302    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:37:28.552307 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:28.552262    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a08570b-9wfk6"]
Oct 30 17:37:28.552942 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:28.552920    4433 cpu_manager.go:414] "containerMap: removing container" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main"
Oct 30 17:37:28.552942 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:28.552942    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main"
Oct 30 17:37:28.553100 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:28.552954    4433 cpu_manager.go:414] "containerMap: removing container" podUID="e1d18482-74d1-4fe0-ae9c-08d49509ae7b" containerName="revoke-keys"
Oct 30 17:37:28.553100 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:28.552960    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1d18482-74d1-4fe0-ae9c-08d49509ae7b" containerName="revoke-keys"
Oct 30 17:37:28.553100 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:28.552967    4433 cpu_manager.go:414] "containerMap: removing container" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main"
Oct 30 17:37:28.553100 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:28.552972    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main"
Oct 30 17:37:28.553100 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:28.552980    4433 cpu_manager.go:414] "containerMap: removing container" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main"
Oct 30 17:37:28.553100 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:28.552984    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main"
Oct 30 17:37:28.553100 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:28.553006    4433 cpu_manager.go:414] "containerMap: removing container" podUID="ea8fea41-d6c6-414b-8a0d-770c435d514c" containerName="istio-proxy"
Oct 30 17:37:28.553100 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:28.553012    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea8fea41-d6c6-414b-8a0d-770c435d514c" containerName="istio-proxy"
Oct 30 17:37:28.553100 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:28.553026    4433 cpu_manager.go:414] "containerMap: removing container" podUID="98a7e2bc-f103-44c3-b5a4-4f93647a28e7" containerName="istio-proxy"
Oct 30 17:37:28.553100 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:28.553030    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98a7e2bc-f103-44c3-b5a4-4f93647a28e7" containerName="istio-proxy"
Oct 30 17:37:28.553836 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:28.553113    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e1669d0-fffa-471f-9275-de0fa70d98e7" containerName="main"
Oct 30 17:37:28.553836 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:28.553127    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea8fea41-d6c6-414b-8a0d-770c435d514c" containerName="istio-proxy"
Oct 30 17:37:28.553836 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:28.553138    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="98a7e2bc-f103-44c3-b5a4-4f93647a28e7" containerName="istio-proxy"
Oct 30 17:37:28.553836 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:28.553150    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab427c7b-928a-4efd-a355-6633379fc574" containerName="main"
Oct 30 17:37:28.553836 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:28.553155    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1d18482-74d1-4fe0-ae9c-08d49509ae7b" containerName="revoke-keys"
Oct 30 17:37:28.553836 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:28.553161    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="6409a840-87ab-4b90-9955-a2a05c47b50f" containerName="main"
Oct 30 17:37:28.558309 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:28.558285    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a08570b-9wfk6"
Oct 30 17:37:28.576929 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:28.576902    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a08570b-9wfk6"]
Oct 30 17:37:28.649992 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:28.649963    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:37:28.650168 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:28.650147    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\""
Oct 30 17:37:28.682571 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:28.682539    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptl97\" (UniqueName: \"kubernetes.io/projected/20dc57cb-856d-4cc9-876e-cfe562599529-kube-api-access-ptl97\") pod \"maas-api-revoke-sub-2a08570b-9wfk6\" (UID: \"20dc57cb-856d-4cc9-876e-cfe562599529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a08570b-9wfk6"
Oct 30 17:37:28.682571 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:28.682582    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20dc57cb-856d-4cc9-876e-cfe562599529-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a08570b-9wfk6\" (UID: \"20dc57cb-856d-4cc9-876e-cfe562599529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a08570b-9wfk6"
Oct 30 17:37:28.682788 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:28.682606    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20dc57cb-856d-4cc9-876e-cfe562599529-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a08570b-9wfk6\" (UID: \"20dc57cb-856d-4cc9-876e-cfe562599529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a08570b-9wfk6"
Oct 30 17:37:28.784178 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:28.784140    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptl97\" (UniqueName: \"kubernetes.io/projected/20dc57cb-856d-4cc9-876e-cfe562599529-kube-api-access-ptl97\") pod \"maas-api-revoke-sub-2a08570b-9wfk6\" (UID: \"20dc57cb-856d-4cc9-876e-cfe562599529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a08570b-9wfk6"
Oct 30 17:37:28.784378 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:28.784190    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20dc57cb-856d-4cc9-876e-cfe562599529-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a08570b-9wfk6\" (UID: \"20dc57cb-856d-4cc9-876e-cfe562599529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a08570b-9wfk6"
Oct 30 17:37:28.784378 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:28.784218    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20dc57cb-856d-4cc9-876e-cfe562599529-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a08570b-9wfk6\" (UID: \"20dc57cb-856d-4cc9-876e-cfe562599529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a08570b-9wfk6"
Oct 30 17:37:28.786658 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:28.785596    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20dc57cb-856d-4cc9-876e-cfe562599529-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a08570b-9wfk6\" (UID: \"20dc57cb-856d-4cc9-876e-cfe562599529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a08570b-9wfk6"
Oct 30 17:37:28.788428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:28.787608    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20dc57cb-856d-4cc9-876e-cfe562599529-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a08570b-9wfk6\" (UID: \"20dc57cb-856d-4cc9-876e-cfe562599529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a08570b-9wfk6"
Oct 30 17:37:28.807021 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:28.806958    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptl97\" (UniqueName: \"kubernetes.io/projected/20dc57cb-856d-4cc9-876e-cfe562599529-kube-api-access-ptl97\") pod \"maas-api-revoke-sub-2a08570b-9wfk6\" (UID: \"20dc57cb-856d-4cc9-876e-cfe562599529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a08570b-9wfk6"
Oct 30 17:37:28.872662 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:28.872628    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a08570b-9wfk6"
Oct 30 17:37:29.180989 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:37:29.180954    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20dc57cb_856d_4cc9_876e_cfe562599529.slice/crio-679d491677fcd9c0f312d05b4c5017c527ae5e890d695657c7d70092efbe73f3 WatchSource:0}: Error finding container 679d491677fcd9c0f312d05b4c5017c527ae5e890d695657c7d70092efbe73f3: Status 404 returned error can't find the container with id 679d491677fcd9c0f312d05b4c5017c527ae5e890d695657c7d70092efbe73f3
Oct 30 17:37:29.181650 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:29.181625    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a08570b-9wfk6"]
Oct 30 17:37:29.308689 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:29.308652    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a08570b-9wfk6" event={"ID":"20dc57cb-856d-4cc9-876e-cfe562599529","Type":"ContainerStarted","Data":"679d491677fcd9c0f312d05b4c5017c527ae5e890d695657c7d70092efbe73f3"}
Oct 30 17:37:29.540582 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:29.540539    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"]
Oct 30 17:37:29.540830 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:29.540801    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49" podUID="df7cd3d7-ced4-415b-b6c8-eb31a545daac" containerName="istio-proxy" containerID="cri-o://9e0a905a9af9d908dd5db3983ede28c64435be767404fe12d61fc5fd11ed4e95" gracePeriod=30
Oct 30 17:37:30.317847 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:30.317813    4433 generic.go:356] "Generic (PLEG): container finished" podID="20dc57cb-856d-4cc9-876e-cfe562599529" containerID="5d4e7ca2fb7b549882ace77606f112f44107e2a4669635db496f5ed0705adea3" exitCode=0
Oct 30 17:37:30.318251 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:30.317904    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a08570b-9wfk6" event={"ID":"20dc57cb-856d-4cc9-876e-cfe562599529","Type":"ContainerDied","Data":"5d4e7ca2fb7b549882ace77606f112f44107e2a4669635db496f5ed0705adea3"}
Oct 30 17:37:31.470558 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:31.470531    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a08570b-9wfk6"
Oct 30 17:37:31.617983 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:31.617895    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20dc57cb-856d-4cc9-876e-cfe562599529-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20dc57cb-856d-4cc9-876e-cfe562599529-maas-api-openshift-ca\") pod \"20dc57cb-856d-4cc9-876e-cfe562599529\" (UID: \"20dc57cb-856d-4cc9-876e-cfe562599529\") "
Oct 30 17:37:31.618136 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:31.618087    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20dc57cb-856d-4cc9-876e-cfe562599529-kube-api-access-ptl97\" (UniqueName: \"kubernetes.io/projected/20dc57cb-856d-4cc9-876e-cfe562599529-kube-api-access-ptl97\") pod \"20dc57cb-856d-4cc9-876e-cfe562599529\" (UID: \"20dc57cb-856d-4cc9-876e-cfe562599529\") "
Oct 30 17:37:31.618212 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:31.618165    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20dc57cb-856d-4cc9-876e-cfe562599529-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20dc57cb-856d-4cc9-876e-cfe562599529-maas-api-xks-ca\") pod \"20dc57cb-856d-4cc9-876e-cfe562599529\" (UID: \"20dc57cb-856d-4cc9-876e-cfe562599529\") "
Oct 30 17:37:31.618357 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:31.618333    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20dc57cb-856d-4cc9-876e-cfe562599529-maas-api-openshift-ca" pod "20dc57cb-856d-4cc9-876e-cfe562599529" (UID: "20dc57cb-856d-4cc9-876e-cfe562599529"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:37:31.618628 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:31.618610    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20dc57cb-856d-4cc9-876e-cfe562599529-maas-api-openshift-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:31.620336 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:31.620303    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20dc57cb-856d-4cc9-876e-cfe562599529-kube-api-access-ptl97" pod "20dc57cb-856d-4cc9-876e-cfe562599529" (UID: "20dc57cb-856d-4cc9-876e-cfe562599529"). InnerVolumeSpecName "kube-api-access-ptl97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:37:31.620475 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:31.620315    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20dc57cb-856d-4cc9-876e-cfe562599529-maas-api-xks-ca" pod "20dc57cb-856d-4cc9-876e-cfe562599529" (UID: "20dc57cb-856d-4cc9-876e-cfe562599529"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:37:31.719505 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:31.719474    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptl97\" (UniqueName: \"kubernetes.io/projected/20dc57cb-856d-4cc9-876e-cfe562599529-kube-api-access-ptl97\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:31.719505 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:31.719503    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20dc57cb-856d-4cc9-876e-cfe562599529-maas-api-xks-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:31.962148 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:31.962112    4433 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:37:31.962332 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:31.962167    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49" podUID="df7cd3d7-ced4-415b-b6c8-eb31a545daac" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:37:32.338064 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:32.337966    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a08570b-9wfk6"
Oct 30 17:37:32.338236 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:32.337987    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a08570b-9wfk6" event={"ID":"20dc57cb-856d-4cc9-876e-cfe562599529","Type":"ContainerDied","Data":"679d491677fcd9c0f312d05b4c5017c527ae5e890d695657c7d70092efbe73f3"}
Oct 30 17:37:32.338236 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:32.338155    4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="679d491677fcd9c0f312d05b4c5017c527ae5e890d695657c7d70092efbe73f3"
Oct 30 17:37:32.370191 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:32.370160    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"]
Oct 30 17:37:32.370386 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:32.370364    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m" podUID="8b410f33-6553-4f8d-972f-b21558c4db1a" containerName="istio-proxy" containerID="cri-o://9486e56ff5604bb9dc985aedae9cceb6fe908780dac53f09b8b5df8b1d2cc350" gracePeriod=30
Oct 30 17:37:32.553755 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:32.553694    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"]
Oct 30 17:37:32.554560 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:32.554525    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt" podUID="1a71489d-ed10-4d2e-93df-d5770efaf301" containerName="istio-proxy" containerID="cri-o://4484f45df8f6f8358feb8d48a598cce5355e81ac1b1e0e7f1d07f0d1562f2210" gracePeriod=30
Oct 30 17:37:34.400935 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:34.400896    4433 patch_prober.go:29] interesting pod/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:37:34.401337 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:34.400958    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m" podUID="8b410f33-6553-4f8d-972f-b21558c4db1a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:37:34.817070 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:34.817035    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"
Oct 30 17:37:34.956341 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:34.956296    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df7cd3d7-ced4-415b-b6c8-eb31a545daac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df7cd3d7-ced4-415b-b6c8-eb31a545daac-istiod-ca-cert\") pod \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") "
Oct 30 17:37:34.956341 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:34.956348    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df7cd3d7-ced4-415b-b6c8-eb31a545daac-kube-api-access-bk87d\" (UniqueName: \"kubernetes.io/projected/df7cd3d7-ced4-415b-b6c8-eb31a545daac-kube-api-access-bk87d\") pod \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") "
Oct 30 17:37:34.956645 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:34.956377    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df7cd3d7-ced4-415b-b6c8-eb31a545daac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7cd3d7-ced4-415b-b6c8-eb31a545daac-workload-socket\") pod \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") "
Oct 30 17:37:34.956645 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:34.956442    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df7cd3d7-ced4-415b-b6c8-eb31a545daac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df7cd3d7-ced4-415b-b6c8-eb31a545daac-istio-podinfo\") pod \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") "
Oct 30 17:37:34.956645 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:34.956510    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df7cd3d7-ced4-415b-b6c8-eb31a545daac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df7cd3d7-ced4-415b-b6c8-eb31a545daac-istio-data\") pod \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") "
Oct 30 17:37:34.956645 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:34.956557    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df7cd3d7-ced4-415b-b6c8-eb31a545daac-istio-token\" (UniqueName: \"kubernetes.io/projected/df7cd3d7-ced4-415b-b6c8-eb31a545daac-istio-token\") pod \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") "
Oct 30 17:37:34.956867 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:34.956700    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df7cd3d7-ced4-415b-b6c8-eb31a545daac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df7cd3d7-ced4-415b-b6c8-eb31a545daac-istio-envoy\") pod \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") "
Oct 30 17:37:34.956867 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:34.956742    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df7cd3d7-ced4-415b-b6c8-eb31a545daac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df7cd3d7-ced4-415b-b6c8-eb31a545daac-workload-certs\") pod \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") "
Oct 30 17:37:34.956867 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:34.956747    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7cd3d7-ced4-415b-b6c8-eb31a545daac-workload-socket" pod "df7cd3d7-ced4-415b-b6c8-eb31a545daac" (UID: "df7cd3d7-ced4-415b-b6c8-eb31a545daac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:37:34.956867 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:34.956770    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df7cd3d7-ced4-415b-b6c8-eb31a545daac-istiod-ca-cert" pod "df7cd3d7-ced4-415b-b6c8-eb31a545daac" (UID: "df7cd3d7-ced4-415b-b6c8-eb31a545daac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:37:34.957100 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:34.956887    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7cd3d7-ced4-415b-b6c8-eb31a545daac-istio-data" pod "df7cd3d7-ced4-415b-b6c8-eb31a545daac" (UID: "df7cd3d7-ced4-415b-b6c8-eb31a545daac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:37:34.957100 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:34.956779    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df7cd3d7-ced4-415b-b6c8-eb31a545daac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7cd3d7-ced4-415b-b6c8-eb31a545daac-credential-socket\") pod \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\" (UID: \"df7cd3d7-ced4-415b-b6c8-eb31a545daac\") "
Oct 30 17:37:34.957100 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:34.957042    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7cd3d7-ced4-415b-b6c8-eb31a545daac-credential-socket" pod "df7cd3d7-ced4-415b-b6c8-eb31a545daac" (UID: "df7cd3d7-ced4-415b-b6c8-eb31a545daac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:37:34.957234 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:34.957102    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7cd3d7-ced4-415b-b6c8-eb31a545daac-workload-certs" pod "df7cd3d7-ced4-415b-b6c8-eb31a545daac" (UID: "df7cd3d7-ced4-415b-b6c8-eb31a545daac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:37:34.957443 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:34.957420    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df7cd3d7-ced4-415b-b6c8-eb31a545daac-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:34.957511 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:34.957446    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7cd3d7-ced4-415b-b6c8-eb31a545daac-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:34.957511 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:34.957460    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df7cd3d7-ced4-415b-b6c8-eb31a545daac-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:34.957511 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:34.957475    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7cd3d7-ced4-415b-b6c8-eb31a545daac-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:34.957511 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:34.957488    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df7cd3d7-ced4-415b-b6c8-eb31a545daac-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:34.958820 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:34.958789    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df7cd3d7-ced4-415b-b6c8-eb31a545daac-istio-podinfo" pod "df7cd3d7-ced4-415b-b6c8-eb31a545daac" (UID: "df7cd3d7-ced4-415b-b6c8-eb31a545daac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:37:34.958920 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:34.958836    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df7cd3d7-ced4-415b-b6c8-eb31a545daac-istio-token" pod "df7cd3d7-ced4-415b-b6c8-eb31a545daac" (UID: "df7cd3d7-ced4-415b-b6c8-eb31a545daac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:37:34.959298 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:34.959272    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7cd3d7-ced4-415b-b6c8-eb31a545daac-istio-envoy" pod "df7cd3d7-ced4-415b-b6c8-eb31a545daac" (UID: "df7cd3d7-ced4-415b-b6c8-eb31a545daac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:37:34.959298 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:34.959283    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df7cd3d7-ced4-415b-b6c8-eb31a545daac-kube-api-access-bk87d" pod "df7cd3d7-ced4-415b-b6c8-eb31a545daac" (UID: "df7cd3d7-ced4-415b-b6c8-eb31a545daac"). InnerVolumeSpecName "kube-api-access-bk87d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:37:35.059050 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:35.059004    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk87d\" (UniqueName: \"kubernetes.io/projected/df7cd3d7-ced4-415b-b6c8-eb31a545daac-kube-api-access-bk87d\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:35.059050 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:35.059047    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df7cd3d7-ced4-415b-b6c8-eb31a545daac-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:35.059050 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:35.059058    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df7cd3d7-ced4-415b-b6c8-eb31a545daac-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:35.059304 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:35.059066    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df7cd3d7-ced4-415b-b6c8-eb31a545daac-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:35.358843 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:35.358748    4433 generic.go:356] "Generic (PLEG): container finished" podID="df7cd3d7-ced4-415b-b6c8-eb31a545daac" containerID="9e0a905a9af9d908dd5db3983ede28c64435be767404fe12d61fc5fd11ed4e95" exitCode=0
Oct 30 17:37:35.358843 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:35.358794    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"
Oct 30 17:37:35.358843 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:35.358828    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49" event={"ID":"df7cd3d7-ced4-415b-b6c8-eb31a545daac","Type":"ContainerDied","Data":"9e0a905a9af9d908dd5db3983ede28c64435be767404fe12d61fc5fd11ed4e95"}
Oct 30 17:37:35.359111 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:35.358884    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49" event={"ID":"df7cd3d7-ced4-415b-b6c8-eb31a545daac","Type":"ContainerDied","Data":"98ceba1597476f87e15300ecc411eff5cb1fb13e0abcf0840ad7824c1683c328"}
Oct 30 17:37:35.359111 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:35.358909    4433 scope.go:122] "RemoveContainer" containerID="9e0a905a9af9d908dd5db3983ede28c64435be767404fe12d61fc5fd11ed4e95"
Oct 30 17:37:35.369779 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:35.369758    4433 scope.go:122] "RemoveContainer" containerID="9e0a905a9af9d908dd5db3983ede28c64435be767404fe12d61fc5fd11ed4e95"
Oct 30 17:37:35.370075 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:37:35.370054    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e0a905a9af9d908dd5db3983ede28c64435be767404fe12d61fc5fd11ed4e95\": container with ID starting with 9e0a905a9af9d908dd5db3983ede28c64435be767404fe12d61fc5fd11ed4e95 not found: ID does not exist" containerID="9e0a905a9af9d908dd5db3983ede28c64435be767404fe12d61fc5fd11ed4e95"
Oct 30 17:37:35.370171 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:35.370087    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e0a905a9af9d908dd5db3983ede28c64435be767404fe12d61fc5fd11ed4e95"} err="failed to get container status \"9e0a905a9af9d908dd5db3983ede28c64435be767404fe12d61fc5fd11ed4e95\": rpc error: code = NotFound desc = could not find container \"9e0a905a9af9d908dd5db3983ede28c64435be767404fe12d61fc5fd11ed4e95\": container with ID starting with 9e0a905a9af9d908dd5db3983ede28c64435be767404fe12d61fc5fd11ed4e95 not found: ID does not exist"
Oct 30 17:37:35.424586 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:35.424543    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"]
Oct 30 17:37:35.435754 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:35.435723    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-bgn49"]
Oct 30 17:37:35.694567 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:35.694535    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df7cd3d7-ced4-415b-b6c8-eb31a545daac" path="/var/lib/kubelet/pods/df7cd3d7-ced4-415b-b6c8-eb31a545daac/volumes"
Oct 30 17:37:37.695908 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.695875    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:37:37.788151 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.788104    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b410f33-6553-4f8d-972f-b21558c4db1a-kube-api-access-b86wj\" (UniqueName: \"kubernetes.io/projected/8b410f33-6553-4f8d-972f-b21558c4db1a-kube-api-access-b86wj\") pod \"8b410f33-6553-4f8d-972f-b21558c4db1a\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") "
Oct 30 17:37:37.788349 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.788167    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b410f33-6553-4f8d-972f-b21558c4db1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b410f33-6553-4f8d-972f-b21558c4db1a-credential-socket\") pod \"8b410f33-6553-4f8d-972f-b21558c4db1a\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") "
Oct 30 17:37:37.788349 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.788281    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b410f33-6553-4f8d-972f-b21558c4db1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b410f33-6553-4f8d-972f-b21558c4db1a-workload-certs\") pod \"8b410f33-6553-4f8d-972f-b21558c4db1a\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") "
Oct 30 17:37:37.788349 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.788342    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b410f33-6553-4f8d-972f-b21558c4db1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b410f33-6553-4f8d-972f-b21558c4db1a-istio-podinfo\") pod \"8b410f33-6553-4f8d-972f-b21558c4db1a\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") "
Oct 30 17:37:37.788590 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.788388    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b410f33-6553-4f8d-972f-b21558c4db1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b410f33-6553-4f8d-972f-b21558c4db1a-workload-socket\") pod \"8b410f33-6553-4f8d-972f-b21558c4db1a\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") "
Oct 30 17:37:37.788590 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.788505    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b410f33-6553-4f8d-972f-b21558c4db1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b410f33-6553-4f8d-972f-b21558c4db1a-istio-envoy\") pod \"8b410f33-6553-4f8d-972f-b21558c4db1a\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") "
Oct 30 17:37:37.788590 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.788546    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b410f33-6553-4f8d-972f-b21558c4db1a-credential-socket" pod "8b410f33-6553-4f8d-972f-b21558c4db1a" (UID: "8b410f33-6553-4f8d-972f-b21558c4db1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:37:37.788590 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.788570    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b410f33-6553-4f8d-972f-b21558c4db1a-istio-token\" (UniqueName: \"kubernetes.io/projected/8b410f33-6553-4f8d-972f-b21558c4db1a-istio-token\") pod \"8b410f33-6553-4f8d-972f-b21558c4db1a\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") "
Oct 30 17:37:37.788824 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.788644    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b410f33-6553-4f8d-972f-b21558c4db1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b410f33-6553-4f8d-972f-b21558c4db1a-istio-data\") pod \"8b410f33-6553-4f8d-972f-b21558c4db1a\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") "
Oct 30 17:37:37.788824 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.788713    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b410f33-6553-4f8d-972f-b21558c4db1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b410f33-6553-4f8d-972f-b21558c4db1a-istiod-ca-cert\") pod \"8b410f33-6553-4f8d-972f-b21558c4db1a\" (UID: \"8b410f33-6553-4f8d-972f-b21558c4db1a\") "
Oct 30 17:37:37.789203 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.789175    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b410f33-6553-4f8d-972f-b21558c4db1a-istio-data" pod "8b410f33-6553-4f8d-972f-b21558c4db1a" (UID: "8b410f33-6553-4f8d-972f-b21558c4db1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:37:37.789423 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.789385    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b410f33-6553-4f8d-972f-b21558c4db1a-workload-socket" pod "8b410f33-6553-4f8d-972f-b21558c4db1a" (UID: "8b410f33-6553-4f8d-972f-b21558c4db1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:37:37.789697 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.789674    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b410f33-6553-4f8d-972f-b21558c4db1a-workload-certs" pod "8b410f33-6553-4f8d-972f-b21558c4db1a" (UID: "8b410f33-6553-4f8d-972f-b21558c4db1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:37:37.789783 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.789756    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b410f33-6553-4f8d-972f-b21558c4db1a-istiod-ca-cert" pod "8b410f33-6553-4f8d-972f-b21558c4db1a" (UID: "8b410f33-6553-4f8d-972f-b21558c4db1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:37:37.790003 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.789978    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b410f33-6553-4f8d-972f-b21558c4db1a-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:37.790086 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.790010    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b410f33-6553-4f8d-972f-b21558c4db1a-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:37.790086 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.790027    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b410f33-6553-4f8d-972f-b21558c4db1a-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:37.790086 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.790041    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b410f33-6553-4f8d-972f-b21558c4db1a-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:37.790086 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.790055    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b410f33-6553-4f8d-972f-b21558c4db1a-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:37.791225 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.791195    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b410f33-6553-4f8d-972f-b21558c4db1a-kube-api-access-b86wj" pod "8b410f33-6553-4f8d-972f-b21558c4db1a" (UID: "8b410f33-6553-4f8d-972f-b21558c4db1a"). InnerVolumeSpecName "kube-api-access-b86wj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:37:37.791776 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.791746    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b410f33-6553-4f8d-972f-b21558c4db1a-istio-token" pod "8b410f33-6553-4f8d-972f-b21558c4db1a" (UID: "8b410f33-6553-4f8d-972f-b21558c4db1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:37:37.792801 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.792774    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b410f33-6553-4f8d-972f-b21558c4db1a-istio-podinfo" pod "8b410f33-6553-4f8d-972f-b21558c4db1a" (UID: "8b410f33-6553-4f8d-972f-b21558c4db1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:37:37.792972 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.792948    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b410f33-6553-4f8d-972f-b21558c4db1a-istio-envoy" pod "8b410f33-6553-4f8d-972f-b21558c4db1a" (UID: "8b410f33-6553-4f8d-972f-b21558c4db1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:37:37.856373 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.856344    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"
Oct 30 17:37:37.890985 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.890939    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b86wj\" (UniqueName: \"kubernetes.io/projected/8b410f33-6553-4f8d-972f-b21558c4db1a-kube-api-access-b86wj\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:37.890985 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.890971    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b410f33-6553-4f8d-972f-b21558c4db1a-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:37.890985 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.890980    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b410f33-6553-4f8d-972f-b21558c4db1a-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:37.891320 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.891000    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b410f33-6553-4f8d-972f-b21558c4db1a-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:37.992185 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.992147    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a71489d-ed10-4d2e-93df-d5770efaf301-istio-token\" (UniqueName: \"kubernetes.io/projected/1a71489d-ed10-4d2e-93df-d5770efaf301-istio-token\") pod \"1a71489d-ed10-4d2e-93df-d5770efaf301\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") "
Oct 30 17:37:37.992374 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.992196    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a71489d-ed10-4d2e-93df-d5770efaf301-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a71489d-ed10-4d2e-93df-d5770efaf301-workload-certs\") pod \"1a71489d-ed10-4d2e-93df-d5770efaf301\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") "
Oct 30 17:37:37.992374 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.992312    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a71489d-ed10-4d2e-93df-d5770efaf301-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a71489d-ed10-4d2e-93df-d5770efaf301-istio-podinfo\") pod \"1a71489d-ed10-4d2e-93df-d5770efaf301\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") "
Oct 30 17:37:37.992374 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.992345    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a71489d-ed10-4d2e-93df-d5770efaf301-kube-api-access-qq4b4\" (UniqueName: \"kubernetes.io/projected/1a71489d-ed10-4d2e-93df-d5770efaf301-kube-api-access-qq4b4\") pod \"1a71489d-ed10-4d2e-93df-d5770efaf301\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") "
Oct 30 17:37:37.992638 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.992375    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a71489d-ed10-4d2e-93df-d5770efaf301-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a71489d-ed10-4d2e-93df-d5770efaf301-credential-socket\") pod \"1a71489d-ed10-4d2e-93df-d5770efaf301\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") "
Oct 30 17:37:37.992638 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.992537    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a71489d-ed10-4d2e-93df-d5770efaf301-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a71489d-ed10-4d2e-93df-d5770efaf301-istio-data\") pod \"1a71489d-ed10-4d2e-93df-d5770efaf301\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") "
Oct 30 17:37:37.992638 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.992591    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a71489d-ed10-4d2e-93df-d5770efaf301-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a71489d-ed10-4d2e-93df-d5770efaf301-istio-envoy\") pod \"1a71489d-ed10-4d2e-93df-d5770efaf301\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") "
Oct 30 17:37:37.992804 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.992612    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a71489d-ed10-4d2e-93df-d5770efaf301-workload-certs" pod "1a71489d-ed10-4d2e-93df-d5770efaf301" (UID: "1a71489d-ed10-4d2e-93df-d5770efaf301"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:37:37.992804 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.992627    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a71489d-ed10-4d2e-93df-d5770efaf301-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a71489d-ed10-4d2e-93df-d5770efaf301-istiod-ca-cert\") pod \"1a71489d-ed10-4d2e-93df-d5770efaf301\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") "
Oct 30 17:37:37.992804 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.992703    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a71489d-ed10-4d2e-93df-d5770efaf301-credential-socket" pod "1a71489d-ed10-4d2e-93df-d5770efaf301" (UID: "1a71489d-ed10-4d2e-93df-d5770efaf301"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:37:37.992804 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.992758    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a71489d-ed10-4d2e-93df-d5770efaf301-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a71489d-ed10-4d2e-93df-d5770efaf301-workload-socket\") pod \"1a71489d-ed10-4d2e-93df-d5770efaf301\" (UID: \"1a71489d-ed10-4d2e-93df-d5770efaf301\") "
Oct 30 17:37:37.992804 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.992792    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a71489d-ed10-4d2e-93df-d5770efaf301-istio-data" pod "1a71489d-ed10-4d2e-93df-d5770efaf301" (UID: "1a71489d-ed10-4d2e-93df-d5770efaf301"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:37:37.993088 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.992987    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a71489d-ed10-4d2e-93df-d5770efaf301-istiod-ca-cert" pod "1a71489d-ed10-4d2e-93df-d5770efaf301" (UID: "1a71489d-ed10-4d2e-93df-d5770efaf301"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:37:37.993141 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.993081    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a71489d-ed10-4d2e-93df-d5770efaf301-workload-socket" pod "1a71489d-ed10-4d2e-93df-d5770efaf301" (UID: "1a71489d-ed10-4d2e-93df-d5770efaf301"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:37:37.993360 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.993336    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a71489d-ed10-4d2e-93df-d5770efaf301-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:37.993451 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.993367    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a71489d-ed10-4d2e-93df-d5770efaf301-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:37.993451 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.993385    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a71489d-ed10-4d2e-93df-d5770efaf301-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:37.993451 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.993441    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a71489d-ed10-4d2e-93df-d5770efaf301-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:37.993569 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.993457    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a71489d-ed10-4d2e-93df-d5770efaf301-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:37.994692 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.994668    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a71489d-ed10-4d2e-93df-d5770efaf301-istio-podinfo" pod "1a71489d-ed10-4d2e-93df-d5770efaf301" (UID: "1a71489d-ed10-4d2e-93df-d5770efaf301"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:37:37.994792 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.994765    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a71489d-ed10-4d2e-93df-d5770efaf301-kube-api-access-qq4b4" pod "1a71489d-ed10-4d2e-93df-d5770efaf301" (UID: "1a71489d-ed10-4d2e-93df-d5770efaf301"). InnerVolumeSpecName "kube-api-access-qq4b4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:37:37.994895 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.994878    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a71489d-ed10-4d2e-93df-d5770efaf301-istio-token" pod "1a71489d-ed10-4d2e-93df-d5770efaf301" (UID: "1a71489d-ed10-4d2e-93df-d5770efaf301"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:37:37.994985 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:37.994965    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a71489d-ed10-4d2e-93df-d5770efaf301-istio-envoy" pod "1a71489d-ed10-4d2e-93df-d5770efaf301" (UID: "1a71489d-ed10-4d2e-93df-d5770efaf301"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:37:38.094470 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:38.094427    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a71489d-ed10-4d2e-93df-d5770efaf301-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:38.094470 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:38.094468    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a71489d-ed10-4d2e-93df-d5770efaf301-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:38.094683 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:38.094485    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq4b4\" (UniqueName: \"kubernetes.io/projected/1a71489d-ed10-4d2e-93df-d5770efaf301-kube-api-access-qq4b4\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:38.094683 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:38.094502    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a71489d-ed10-4d2e-93df-d5770efaf301-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:37:38.380947 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:38.380858    4433 generic.go:356] "Generic (PLEG): container finished" podID="8b410f33-6553-4f8d-972f-b21558c4db1a" containerID="9486e56ff5604bb9dc985aedae9cceb6fe908780dac53f09b8b5df8b1d2cc350" exitCode=0
Oct 30 17:37:38.380947 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:38.380909    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"
Oct 30 17:37:38.380947 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:38.380936    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m" event={"ID":"8b410f33-6553-4f8d-972f-b21558c4db1a","Type":"ContainerDied","Data":"9486e56ff5604bb9dc985aedae9cceb6fe908780dac53f09b8b5df8b1d2cc350"}
Oct 30 17:37:38.381470 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:38.380992    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m" event={"ID":"8b410f33-6553-4f8d-972f-b21558c4db1a","Type":"ContainerDied","Data":"d9e523a180dd947792761ad9a52f79b9b451c8c29b721479e05b63181f9190a4"}
Oct 30 17:37:38.381470 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:38.381020    4433 scope.go:122] "RemoveContainer" containerID="9486e56ff5604bb9dc985aedae9cceb6fe908780dac53f09b8b5df8b1d2cc350"
Oct 30 17:37:38.382380 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:38.382331    4433 generic.go:356] "Generic (PLEG): container finished" podID="1a71489d-ed10-4d2e-93df-d5770efaf301" containerID="4484f45df8f6f8358feb8d48a598cce5355e81ac1b1e0e7f1d07f0d1562f2210" exitCode=0
Oct 30 17:37:38.382468 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:38.382387    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"
Oct 30 17:37:38.382468 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:38.382420    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt" event={"ID":"1a71489d-ed10-4d2e-93df-d5770efaf301","Type":"ContainerDied","Data":"4484f45df8f6f8358feb8d48a598cce5355e81ac1b1e0e7f1d07f0d1562f2210"}
Oct 30 17:37:38.382468 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:38.382466    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt" event={"ID":"1a71489d-ed10-4d2e-93df-d5770efaf301","Type":"ContainerDied","Data":"a864a5bc3a1ae189bba195629fa3388081e291dfd5e6841c159ce93389717fd1"}
Oct 30 17:37:38.391564 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:38.391544    4433 scope.go:122] "RemoveContainer" containerID="cc9c266c294d99efd5fd767e042e7c54ef42b46fe7abd8d39d868139a859c176"
Oct 30 17:37:38.410172 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:38.410152    4433 scope.go:122] "RemoveContainer" containerID="9486e56ff5604bb9dc985aedae9cceb6fe908780dac53f09b8b5df8b1d2cc350"
Oct 30 17:37:38.410502 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:37:38.410481    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9486e56ff5604bb9dc985aedae9cceb6fe908780dac53f09b8b5df8b1d2cc350\": container with ID starting with 9486e56ff5604bb9dc985aedae9cceb6fe908780dac53f09b8b5df8b1d2cc350 not found: ID does not exist" containerID="9486e56ff5604bb9dc985aedae9cceb6fe908780dac53f09b8b5df8b1d2cc350"
Oct 30 17:37:38.410556 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:38.410512    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9486e56ff5604bb9dc985aedae9cceb6fe908780dac53f09b8b5df8b1d2cc350"} err="failed to get container status \"9486e56ff5604bb9dc985aedae9cceb6fe908780dac53f09b8b5df8b1d2cc350\": rpc error: code = NotFound desc = could not find container \"9486e56ff5604bb9dc985aedae9cceb6fe908780dac53f09b8b5df8b1d2cc350\": container with ID starting with 9486e56ff5604bb9dc985aedae9cceb6fe908780dac53f09b8b5df8b1d2cc350 not found: ID does not exist"
Oct 30 17:37:38.410556 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:38.410531    4433 scope.go:122] "RemoveContainer" containerID="cc9c266c294d99efd5fd767e042e7c54ef42b46fe7abd8d39d868139a859c176"
Oct 30 17:37:38.410824 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:37:38.410799    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc9c266c294d99efd5fd767e042e7c54ef42b46fe7abd8d39d868139a859c176\": container with ID starting with cc9c266c294d99efd5fd767e042e7c54ef42b46fe7abd8d39d868139a859c176 not found: ID does not exist" containerID="cc9c266c294d99efd5fd767e042e7c54ef42b46fe7abd8d39d868139a859c176"
Oct 30 17:37:38.410933 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:38.410828    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc9c266c294d99efd5fd767e042e7c54ef42b46fe7abd8d39d868139a859c176"} err="failed to get container status \"cc9c266c294d99efd5fd767e042e7c54ef42b46fe7abd8d39d868139a859c176\": rpc error: code = NotFound desc = could not find container \"cc9c266c294d99efd5fd767e042e7c54ef42b46fe7abd8d39d868139a859c176\": container with ID starting with cc9c266c294d99efd5fd767e042e7c54ef42b46fe7abd8d39d868139a859c176 not found: ID does not exist"
Oct 30 17:37:38.410933 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:38.410847    4433 scope.go:122] "RemoveContainer" containerID="4484f45df8f6f8358feb8d48a598cce5355e81ac1b1e0e7f1d07f0d1562f2210"
Oct 30 17:37:38.421337 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:38.421314    4433 scope.go:122] "RemoveContainer" containerID="4484f45df8f6f8358feb8d48a598cce5355e81ac1b1e0e7f1d07f0d1562f2210"
Oct 30 17:37:38.421677 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:37:38.421654    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4484f45df8f6f8358feb8d48a598cce5355e81ac1b1e0e7f1d07f0d1562f2210\": container with ID starting with 4484f45df8f6f8358feb8d48a598cce5355e81ac1b1e0e7f1d07f0d1562f2210 not found: ID does not exist" containerID="4484f45df8f6f8358feb8d48a598cce5355e81ac1b1e0e7f1d07f0d1562f2210"
Oct 30 17:37:38.421733 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:38.421686    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4484f45df8f6f8358feb8d48a598cce5355e81ac1b1e0e7f1d07f0d1562f2210"} err="failed to get container status \"4484f45df8f6f8358feb8d48a598cce5355e81ac1b1e0e7f1d07f0d1562f2210\": rpc error: code = NotFound desc = could not find container \"4484f45df8f6f8358feb8d48a598cce5355e81ac1b1e0e7f1d07f0d1562f2210\": container with ID starting with 4484f45df8f6f8358feb8d48a598cce5355e81ac1b1e0e7f1d07f0d1562f2210 not found: ID does not exist"
Oct 30 17:37:38.449576 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:38.449538    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"]
Oct 30 17:37:38.461278 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:38.461234    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a949e7-openshift-default-f9c48fff7-5287m"]
Oct 30 17:37:38.486257 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:38.486221    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"]
Oct 30 17:37:38.490933 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:38.490900    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-cxwmt"]
Oct 30 17:37:39.694711 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:39.694676    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a71489d-ed10-4d2e-93df-d5770efaf301" path="/var/lib/kubelet/pods/1a71489d-ed10-4d2e-93df-d5770efaf301/volumes"
Oct 30 17:37:39.695173 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:39.695158    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b410f33-6553-4f8d-972f-b21558c4db1a" path="/var/lib/kubelet/pods/8b410f33-6553-4f8d-972f-b21558c4db1a/volumes"
Oct 30 17:37:50.951415 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:37:50.951366    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:37:56.978579 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:37:56.978539    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:38:02.471418 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:02.471357    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"]
Oct 30 17:38:02.471950 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:02.471709    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d" podUID="9f851311-31f7-477c-a37c-024f63ea7fcf" containerName="istio-proxy" containerID="cri-o://e91a0c7cbed7192ffbe2e5fd8edf39d01843d425024883e14867ce49744acbae" gracePeriod=30
Oct 30 17:38:06.036941 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:06.036896    4433 scope.go:122] "RemoveContainer" containerID="f9e2987d8d380f9694fe0b1d9ebad9da12e532595279bf5d5068d7b9489e6f5b"
Oct 30 17:38:07.740385 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:07.740354    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"
Oct 30 17:38:07.855702 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:07.855589    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f851311-31f7-477c-a37c-024f63ea7fcf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f851311-31f7-477c-a37c-024f63ea7fcf-istio-envoy\") pod \"9f851311-31f7-477c-a37c-024f63ea7fcf\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") "
Oct 30 17:38:07.855887 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:07.855724    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f851311-31f7-477c-a37c-024f63ea7fcf-istio-token\" (UniqueName: \"kubernetes.io/projected/9f851311-31f7-477c-a37c-024f63ea7fcf-istio-token\") pod \"9f851311-31f7-477c-a37c-024f63ea7fcf\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") "
Oct 30 17:38:07.855887 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:07.855774    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f851311-31f7-477c-a37c-024f63ea7fcf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f851311-31f7-477c-a37c-024f63ea7fcf-istiod-ca-cert\") pod \"9f851311-31f7-477c-a37c-024f63ea7fcf\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") "
Oct 30 17:38:07.855887 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:07.855851    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f851311-31f7-477c-a37c-024f63ea7fcf-kube-api-access-xrpkl\" (UniqueName: \"kubernetes.io/projected/9f851311-31f7-477c-a37c-024f63ea7fcf-kube-api-access-xrpkl\") pod \"9f851311-31f7-477c-a37c-024f63ea7fcf\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") "
Oct 30 17:38:07.856132 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:07.855910    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f851311-31f7-477c-a37c-024f63ea7fcf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f851311-31f7-477c-a37c-024f63ea7fcf-credential-socket\") pod \"9f851311-31f7-477c-a37c-024f63ea7fcf\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") "
Oct 30 17:38:07.856132 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:07.855990    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f851311-31f7-477c-a37c-024f63ea7fcf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f851311-31f7-477c-a37c-024f63ea7fcf-istio-podinfo\") pod \"9f851311-31f7-477c-a37c-024f63ea7fcf\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") "
Oct 30 17:38:07.856132 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:07.856067    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f851311-31f7-477c-a37c-024f63ea7fcf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f851311-31f7-477c-a37c-024f63ea7fcf-istio-data\") pod \"9f851311-31f7-477c-a37c-024f63ea7fcf\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") "
Oct 30 17:38:07.856803 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:07.856774    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f851311-31f7-477c-a37c-024f63ea7fcf-istiod-ca-cert" pod "9f851311-31f7-477c-a37c-024f63ea7fcf" (UID: "9f851311-31f7-477c-a37c-024f63ea7fcf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:38:07.859135 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:07.858628    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f851311-31f7-477c-a37c-024f63ea7fcf-istio-data" pod "9f851311-31f7-477c-a37c-024f63ea7fcf" (UID: "9f851311-31f7-477c-a37c-024f63ea7fcf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:38:07.859135 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:07.858871    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f851311-31f7-477c-a37c-024f63ea7fcf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f851311-31f7-477c-a37c-024f63ea7fcf-workload-certs\") pod \"9f851311-31f7-477c-a37c-024f63ea7fcf\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") "
Oct 30 17:38:07.859135 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:07.858933    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f851311-31f7-477c-a37c-024f63ea7fcf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f851311-31f7-477c-a37c-024f63ea7fcf-workload-socket\") pod \"9f851311-31f7-477c-a37c-024f63ea7fcf\" (UID: \"9f851311-31f7-477c-a37c-024f63ea7fcf\") "
Oct 30 17:38:07.859553 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:07.859528    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f851311-31f7-477c-a37c-024f63ea7fcf-credential-socket" pod "9f851311-31f7-477c-a37c-024f63ea7fcf" (UID: "9f851311-31f7-477c-a37c-024f63ea7fcf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:38:07.859826 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:07.859795    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f851311-31f7-477c-a37c-024f63ea7fcf-workload-socket" pod "9f851311-31f7-477c-a37c-024f63ea7fcf" (UID: "9f851311-31f7-477c-a37c-024f63ea7fcf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:38:07.859951 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:07.859902    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f851311-31f7-477c-a37c-024f63ea7fcf-workload-certs" pod "9f851311-31f7-477c-a37c-024f63ea7fcf" (UID: "9f851311-31f7-477c-a37c-024f63ea7fcf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:38:07.861320 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:07.860340    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f851311-31f7-477c-a37c-024f63ea7fcf-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:38:07.861320 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:07.860375    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f851311-31f7-477c-a37c-024f63ea7fcf-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:38:07.861320 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:07.860399    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f851311-31f7-477c-a37c-024f63ea7fcf-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:38:07.861320 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:07.860436    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f851311-31f7-477c-a37c-024f63ea7fcf-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:38:07.861320 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:07.860450    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f851311-31f7-477c-a37c-024f63ea7fcf-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:38:07.865040 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:07.862356    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f851311-31f7-477c-a37c-024f63ea7fcf-kube-api-access-xrpkl" pod "9f851311-31f7-477c-a37c-024f63ea7fcf" (UID: "9f851311-31f7-477c-a37c-024f63ea7fcf"). InnerVolumeSpecName "kube-api-access-xrpkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:38:07.865040 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:07.864137    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f851311-31f7-477c-a37c-024f63ea7fcf-istio-podinfo" pod "9f851311-31f7-477c-a37c-024f63ea7fcf" (UID: "9f851311-31f7-477c-a37c-024f63ea7fcf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:38:07.866686 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:07.866653    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f851311-31f7-477c-a37c-024f63ea7fcf-istio-envoy" pod "9f851311-31f7-477c-a37c-024f63ea7fcf" (UID: "9f851311-31f7-477c-a37c-024f63ea7fcf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:38:07.866810 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:07.866791    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f851311-31f7-477c-a37c-024f63ea7fcf-istio-token" pod "9f851311-31f7-477c-a37c-024f63ea7fcf" (UID: "9f851311-31f7-477c-a37c-024f63ea7fcf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:38:07.961257 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:07.961225    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f851311-31f7-477c-a37c-024f63ea7fcf-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:38:07.961257 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:07.961254    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f851311-31f7-477c-a37c-024f63ea7fcf-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:38:07.961257 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:07.961265    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrpkl\" (UniqueName: \"kubernetes.io/projected/9f851311-31f7-477c-a37c-024f63ea7fcf-kube-api-access-xrpkl\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:38:07.961559 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:07.961275    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f851311-31f7-477c-a37c-024f63ea7fcf-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:38:08.244388 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:08.244344    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:38:08.562538 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:08.562441    4433 generic.go:356] "Generic (PLEG): container finished" podID="9f851311-31f7-477c-a37c-024f63ea7fcf" containerID="e91a0c7cbed7192ffbe2e5fd8edf39d01843d425024883e14867ce49744acbae" exitCode=0
Oct 30 17:38:08.562538 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:08.562495    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"
Oct 30 17:38:08.562538 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:08.562530    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d" event={"ID":"9f851311-31f7-477c-a37c-024f63ea7fcf","Type":"ContainerDied","Data":"e91a0c7cbed7192ffbe2e5fd8edf39d01843d425024883e14867ce49744acbae"}
Oct 30 17:38:08.562824 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:08.562578    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d" event={"ID":"9f851311-31f7-477c-a37c-024f63ea7fcf","Type":"ContainerDied","Data":"b5aa664c51cbd0cf3e05ef1f777a4668db5eb44f89b69d4981ae9f20805bedb6"}
Oct 30 17:38:08.562824 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:08.562601    4433 scope.go:122] "RemoveContainer" containerID="e91a0c7cbed7192ffbe2e5fd8edf39d01843d425024883e14867ce49744acbae"
Oct 30 17:38:08.572526 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:08.572465    4433 scope.go:122] "RemoveContainer" containerID="e91a0c7cbed7192ffbe2e5fd8edf39d01843d425024883e14867ce49744acbae"
Oct 30 17:38:08.572858 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:38:08.572838    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e91a0c7cbed7192ffbe2e5fd8edf39d01843d425024883e14867ce49744acbae\": container with ID starting with e91a0c7cbed7192ffbe2e5fd8edf39d01843d425024883e14867ce49744acbae not found: ID does not exist" containerID="e91a0c7cbed7192ffbe2e5fd8edf39d01843d425024883e14867ce49744acbae"
Oct 30 17:38:08.572921 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:08.572866    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e91a0c7cbed7192ffbe2e5fd8edf39d01843d425024883e14867ce49744acbae"} err="failed to get container status \"e91a0c7cbed7192ffbe2e5fd8edf39d01843d425024883e14867ce49744acbae\": rpc error: code = NotFound desc = could not find container \"e91a0c7cbed7192ffbe2e5fd8edf39d01843d425024883e14867ce49744acbae\": container with ID starting with e91a0c7cbed7192ffbe2e5fd8edf39d01843d425024883e14867ce49744acbae not found: ID does not exist"
Oct 30 17:38:08.613338 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:08.613301    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"]
Oct 30 17:38:08.625749 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:08.625719    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-lzt6d"]
Oct 30 17:38:09.695583 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:09.695549    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f851311-31f7-477c-a37c-024f63ea7fcf" path="/var/lib/kubelet/pods/9f851311-31f7-477c-a37c-024f63ea7fcf/volumes"
Oct 30 17:38:24.688713 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:24.688662    4433 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww" secret="" err="secret \"e2e-shared-a-w0-469ad7-openshift-default-dockercfg-q2qkg\" not found"
Oct 30 17:38:24.716534 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:24.715784    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"]
Oct 30 17:38:25.172942 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:25.172855    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:38:25.664516 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:25.664450    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww" podUID="d74c8b0b-844e-493f-a994-5b83b7b73589" containerName="istio-proxy" containerID="cri-o://44e0bc3489b057ec0802fdc99a284108a00a6fbbfa314825aa02d2e6300a09c1" gracePeriod=30
Oct 30 17:38:26.979020 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:38:26.978984    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:38:28.431469 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:28.431431    4433 patch_prober.go:29] interesting pod/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:38:28.431882 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:28.431502    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww" podUID="d74c8b0b-844e-493f-a994-5b83b7b73589" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:38:30.925119 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:30.925097    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:38:31.010303 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.010222    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d74c8b0b-844e-493f-a994-5b83b7b73589-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d74c8b0b-844e-493f-a994-5b83b7b73589-workload-certs\") pod \"d74c8b0b-844e-493f-a994-5b83b7b73589\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") "
Oct 30 17:38:31.010303 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.010292    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d74c8b0b-844e-493f-a994-5b83b7b73589-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d74c8b0b-844e-493f-a994-5b83b7b73589-istiod-ca-cert\") pod \"d74c8b0b-844e-493f-a994-5b83b7b73589\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") "
Oct 30 17:38:31.010707 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.010341    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d74c8b0b-844e-493f-a994-5b83b7b73589-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d74c8b0b-844e-493f-a994-5b83b7b73589-istio-envoy\") pod \"d74c8b0b-844e-493f-a994-5b83b7b73589\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") "
Oct 30 17:38:31.010707 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.010557    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d74c8b0b-844e-493f-a994-5b83b7b73589-workload-certs" pod "d74c8b0b-844e-493f-a994-5b83b7b73589" (UID: "d74c8b0b-844e-493f-a994-5b83b7b73589"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:38:31.010707 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.010580    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d74c8b0b-844e-493f-a994-5b83b7b73589-kube-api-access-mrgbt\" (UniqueName: \"kubernetes.io/projected/d74c8b0b-844e-493f-a994-5b83b7b73589-kube-api-access-mrgbt\") pod \"d74c8b0b-844e-493f-a994-5b83b7b73589\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") "
Oct 30 17:38:31.010707 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.010642    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d74c8b0b-844e-493f-a994-5b83b7b73589-istio-token\" (UniqueName: \"kubernetes.io/projected/d74c8b0b-844e-493f-a994-5b83b7b73589-istio-token\") pod \"d74c8b0b-844e-493f-a994-5b83b7b73589\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") "
Oct 30 17:38:31.010707 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.010681    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d74c8b0b-844e-493f-a994-5b83b7b73589-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d74c8b0b-844e-493f-a994-5b83b7b73589-istio-data\") pod \"d74c8b0b-844e-493f-a994-5b83b7b73589\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") "
Oct 30 17:38:31.011009 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.010759    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d74c8b0b-844e-493f-a994-5b83b7b73589-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d74c8b0b-844e-493f-a994-5b83b7b73589-workload-socket\") pod \"d74c8b0b-844e-493f-a994-5b83b7b73589\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") "
Oct 30 17:38:31.011009 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.010796    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d74c8b0b-844e-493f-a994-5b83b7b73589-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d74c8b0b-844e-493f-a994-5b83b7b73589-credential-socket\") pod \"d74c8b0b-844e-493f-a994-5b83b7b73589\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") "
Oct 30 17:38:31.011009 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.010860    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d74c8b0b-844e-493f-a994-5b83b7b73589-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d74c8b0b-844e-493f-a994-5b83b7b73589-istio-podinfo\") pod \"d74c8b0b-844e-493f-a994-5b83b7b73589\" (UID: \"d74c8b0b-844e-493f-a994-5b83b7b73589\") "
Oct 30 17:38:31.011009 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.010885    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d74c8b0b-844e-493f-a994-5b83b7b73589-istiod-ca-cert" pod "d74c8b0b-844e-493f-a994-5b83b7b73589" (UID: "d74c8b0b-844e-493f-a994-5b83b7b73589"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:38:31.011231 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.011025    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d74c8b0b-844e-493f-a994-5b83b7b73589-istio-data" pod "d74c8b0b-844e-493f-a994-5b83b7b73589" (UID: "d74c8b0b-844e-493f-a994-5b83b7b73589"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:38:31.011231 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.011168    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d74c8b0b-844e-493f-a994-5b83b7b73589-workload-socket" pod "d74c8b0b-844e-493f-a994-5b83b7b73589" (UID: "d74c8b0b-844e-493f-a994-5b83b7b73589"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:38:31.011394 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.011370    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d74c8b0b-844e-493f-a994-5b83b7b73589-credential-socket" pod "d74c8b0b-844e-493f-a994-5b83b7b73589" (UID: "d74c8b0b-844e-493f-a994-5b83b7b73589"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:38:31.011494 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.011376    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d74c8b0b-844e-493f-a994-5b83b7b73589-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:38:31.011494 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.011453    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d74c8b0b-844e-493f-a994-5b83b7b73589-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:38:31.011494 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.011470    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d74c8b0b-844e-493f-a994-5b83b7b73589-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:38:31.011494 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.011487    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d74c8b0b-844e-493f-a994-5b83b7b73589-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:38:31.012818 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.012787    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d74c8b0b-844e-493f-a994-5b83b7b73589-istio-envoy" pod "d74c8b0b-844e-493f-a994-5b83b7b73589" (UID: "d74c8b0b-844e-493f-a994-5b83b7b73589"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:38:31.013294 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.013268    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d74c8b0b-844e-493f-a994-5b83b7b73589-istio-token" pod "d74c8b0b-844e-493f-a994-5b83b7b73589" (UID: "d74c8b0b-844e-493f-a994-5b83b7b73589"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:38:31.013420 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.013367    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d74c8b0b-844e-493f-a994-5b83b7b73589-kube-api-access-mrgbt" pod "d74c8b0b-844e-493f-a994-5b83b7b73589" (UID: "d74c8b0b-844e-493f-a994-5b83b7b73589"). InnerVolumeSpecName "kube-api-access-mrgbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:38:31.013420 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.013374    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d74c8b0b-844e-493f-a994-5b83b7b73589-istio-podinfo" pod "d74c8b0b-844e-493f-a994-5b83b7b73589" (UID: "d74c8b0b-844e-493f-a994-5b83b7b73589"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:38:31.112681 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.112645    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d74c8b0b-844e-493f-a994-5b83b7b73589-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:38:31.112681 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.112675    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d74c8b0b-844e-493f-a994-5b83b7b73589-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:38:31.112681 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.112685    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d74c8b0b-844e-493f-a994-5b83b7b73589-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:38:31.112932 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.112694    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d74c8b0b-844e-493f-a994-5b83b7b73589-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:38:31.112932 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.112703    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrgbt\" (UniqueName: \"kubernetes.io/projected/d74c8b0b-844e-493f-a994-5b83b7b73589-kube-api-access-mrgbt\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:38:31.703571 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.703529    4433 generic.go:356] "Generic (PLEG): container finished" podID="d74c8b0b-844e-493f-a994-5b83b7b73589" containerID="44e0bc3489b057ec0802fdc99a284108a00a6fbbfa314825aa02d2e6300a09c1" exitCode=0
Oct 30 17:38:31.703795 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.703625    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"
Oct 30 17:38:31.703795 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.703700    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww" event={"ID":"d74c8b0b-844e-493f-a994-5b83b7b73589","Type":"ContainerDied","Data":"44e0bc3489b057ec0802fdc99a284108a00a6fbbfa314825aa02d2e6300a09c1"}
Oct 30 17:38:31.703795 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.703746    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww" event={"ID":"d74c8b0b-844e-493f-a994-5b83b7b73589","Type":"ContainerDied","Data":"df0a4718e9be52fd56fd1338d028edb1205652744b1f126c58fe73655fe59109"}
Oct 30 17:38:31.703974 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.703769    4433 scope.go:122] "RemoveContainer" containerID="44e0bc3489b057ec0802fdc99a284108a00a6fbbfa314825aa02d2e6300a09c1"
Oct 30 17:38:31.715968 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.715945    4433 scope.go:122] "RemoveContainer" containerID="890ea6dad27d4f3b9958d2f10ba005dad229051350aa4f19a895e1390d0d8954"
Oct 30 17:38:31.724838 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.724820    4433 scope.go:122] "RemoveContainer" containerID="44e0bc3489b057ec0802fdc99a284108a00a6fbbfa314825aa02d2e6300a09c1"
Oct 30 17:38:31.725141 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:38:31.725122    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44e0bc3489b057ec0802fdc99a284108a00a6fbbfa314825aa02d2e6300a09c1\": container with ID starting with 44e0bc3489b057ec0802fdc99a284108a00a6fbbfa314825aa02d2e6300a09c1 not found: ID does not exist" containerID="44e0bc3489b057ec0802fdc99a284108a00a6fbbfa314825aa02d2e6300a09c1"
Oct 30 17:38:31.725192 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.725153    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44e0bc3489b057ec0802fdc99a284108a00a6fbbfa314825aa02d2e6300a09c1"} err="failed to get container status \"44e0bc3489b057ec0802fdc99a284108a00a6fbbfa314825aa02d2e6300a09c1\": rpc error: code = NotFound desc = could not find container \"44e0bc3489b057ec0802fdc99a284108a00a6fbbfa314825aa02d2e6300a09c1\": container with ID starting with 44e0bc3489b057ec0802fdc99a284108a00a6fbbfa314825aa02d2e6300a09c1 not found: ID does not exist"
Oct 30 17:38:31.725192 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.725172    4433 scope.go:122] "RemoveContainer" containerID="890ea6dad27d4f3b9958d2f10ba005dad229051350aa4f19a895e1390d0d8954"
Oct 30 17:38:31.725430 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:38:31.725392    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"890ea6dad27d4f3b9958d2f10ba005dad229051350aa4f19a895e1390d0d8954\": container with ID starting with 890ea6dad27d4f3b9958d2f10ba005dad229051350aa4f19a895e1390d0d8954 not found: ID does not exist" containerID="890ea6dad27d4f3b9958d2f10ba005dad229051350aa4f19a895e1390d0d8954"
Oct 30 17:38:31.725535 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.725436    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"890ea6dad27d4f3b9958d2f10ba005dad229051350aa4f19a895e1390d0d8954"} err="failed to get container status \"890ea6dad27d4f3b9958d2f10ba005dad229051350aa4f19a895e1390d0d8954\": rpc error: code = NotFound desc = could not find container \"890ea6dad27d4f3b9958d2f10ba005dad229051350aa4f19a895e1390d0d8954\": container with ID starting with 890ea6dad27d4f3b9958d2f10ba005dad229051350aa4f19a895e1390d0d8954 not found: ID does not exist"
Oct 30 17:38:31.766582 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.766547    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"]
Oct 30 17:38:31.777983 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:31.777952    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-zwfww"]
Oct 30 17:38:33.720955 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:33.720917    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d74c8b0b-844e-493f-a994-5b83b7b73589" path="/var/lib/kubelet/pods/d74c8b0b-844e-493f-a994-5b83b7b73589/volumes"
Oct 30 17:38:38.030131 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:38.030088    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e334447d-xlmzn"]
Oct 30 17:38:38.040939 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:38.040910    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e334447d-xlmzn"]
Oct 30 17:38:39.694578 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:39.694540    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fec44098-d2da-43d5-a3da-0667808a27cf" path="/var/lib/kubelet/pods/fec44098-d2da-43d5-a3da-0667808a27cf/volumes"
Oct 30 17:38:42.569231 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:42.569184    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:38:45.626864 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:45.619173    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"]
Oct 30 17:38:45.626864 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:45.619546    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb" podUID="aefefaea-deba-44ec-a357-4fb4ba5e7f6e" containerName="istio-proxy" containerID="cri-o://a424e1c49acb61cdd55378c196749cb7cc6cd98c3aa808c67d9cf4ba91ae229a" gracePeriod=30
Oct 30 17:38:50.818141 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:50.818106    4433 generic.go:356] "Generic (PLEG): container finished" podID="aefefaea-deba-44ec-a357-4fb4ba5e7f6e" containerID="a424e1c49acb61cdd55378c196749cb7cc6cd98c3aa808c67d9cf4ba91ae229a" exitCode=0
Oct 30 17:38:50.818593 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:50.818176    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb" event={"ID":"aefefaea-deba-44ec-a357-4fb4ba5e7f6e","Type":"ContainerDied","Data":"a424e1c49acb61cdd55378c196749cb7cc6cd98c3aa808c67d9cf4ba91ae229a"}
Oct 30 17:38:50.818593 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:50.818270    4433 scope.go:122] "RemoveContainer" containerID="284070c3fc1cda348dc8dd123a9fd8793da2aea27df2249862097a86b81ea93d"
Oct 30 17:38:50.891151 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:50.891092    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:38:50.925949 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:50.925902    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-workload-socket\") pod \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") "
Oct 30 17:38:50.926104 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:50.925972    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-istio-data\") pod \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") "
Oct 30 17:38:50.926104 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:50.926002    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-istio-envoy\") pod \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") "
Oct 30 17:38:50.926104 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:50.926022    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-credential-socket\") pod \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") "
Oct 30 17:38:50.926104 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:50.926069    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-workload-certs\") pod \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") "
Oct 30 17:38:50.926329 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:50.926123    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-istio-token\" (UniqueName: \"kubernetes.io/projected/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-istio-token\") pod \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") "
Oct 30 17:38:50.926329 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:50.926143    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-istiod-ca-cert\") pod \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") "
Oct 30 17:38:50.926329 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:50.926172    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-kube-api-access-42c9j\" (UniqueName: \"kubernetes.io/projected/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-kube-api-access-42c9j\") pod \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") "
Oct 30 17:38:50.926329 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:50.926205    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-istio-podinfo\") pod \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\" (UID: \"aefefaea-deba-44ec-a357-4fb4ba5e7f6e\") "
Oct 30 17:38:50.926329 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:50.926251    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-workload-socket" pod "aefefaea-deba-44ec-a357-4fb4ba5e7f6e" (UID: "aefefaea-deba-44ec-a357-4fb4ba5e7f6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:38:50.926329 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:50.926262    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-istio-data" pod "aefefaea-deba-44ec-a357-4fb4ba5e7f6e" (UID: "aefefaea-deba-44ec-a357-4fb4ba5e7f6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:38:50.926677 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:50.926544    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-credential-socket" pod "aefefaea-deba-44ec-a357-4fb4ba5e7f6e" (UID: "aefefaea-deba-44ec-a357-4fb4ba5e7f6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:38:50.926677 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:50.926560    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-workload-certs" pod "aefefaea-deba-44ec-a357-4fb4ba5e7f6e" (UID: "aefefaea-deba-44ec-a357-4fb4ba5e7f6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:38:50.926677 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:50.926557    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:38:50.926677 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:50.926608    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:38:50.926677 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:50.926618    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-istiod-ca-cert" pod "aefefaea-deba-44ec-a357-4fb4ba5e7f6e" (UID: "aefefaea-deba-44ec-a357-4fb4ba5e7f6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:38:50.928446 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:50.928417    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-istio-token" pod "aefefaea-deba-44ec-a357-4fb4ba5e7f6e" (UID: "aefefaea-deba-44ec-a357-4fb4ba5e7f6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:38:50.928552 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:50.928445    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-istio-podinfo" pod "aefefaea-deba-44ec-a357-4fb4ba5e7f6e" (UID: "aefefaea-deba-44ec-a357-4fb4ba5e7f6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:38:50.928552 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:50.928451    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-istio-envoy" pod "aefefaea-deba-44ec-a357-4fb4ba5e7f6e" (UID: "aefefaea-deba-44ec-a357-4fb4ba5e7f6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:38:50.928633 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:50.928548    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-kube-api-access-42c9j" pod "aefefaea-deba-44ec-a357-4fb4ba5e7f6e" (UID: "aefefaea-deba-44ec-a357-4fb4ba5e7f6e"). InnerVolumeSpecName "kube-api-access-42c9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:38:51.027117 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:51.027081    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:38:51.027117 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:51.027114    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:38:51.027117 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:51.027123    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:38:51.027364 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:51.027132    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:38:51.027364 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:51.027139    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:38:51.027364 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:51.027147    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42c9j\" (UniqueName: \"kubernetes.io/projected/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-kube-api-access-42c9j\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:38:51.027364 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:51.027156    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aefefaea-deba-44ec-a357-4fb4ba5e7f6e-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:38:51.113576 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:51.113536    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"]
Oct 30 17:38:51.825183 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:51.825149    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"
Oct 30 17:38:51.825183 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:51.825177    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb" event={"ID":"aefefaea-deba-44ec-a357-4fb4ba5e7f6e","Type":"ContainerDied","Data":"61868b56164b880d11949e735a23aee3996c2e6dad98bffab2ea29a245be0bfb"}
Oct 30 17:38:51.825740 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:51.825237    4433 scope.go:122] "RemoveContainer" containerID="a424e1c49acb61cdd55378c196749cb7cc6cd98c3aa808c67d9cf4ba91ae229a"
Oct 30 17:38:51.887033 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:51.887000    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"]
Oct 30 17:38:51.890632 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:51.890604    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-gnqkb"]
Oct 30 17:38:53.694048 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:38:53.694016    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aefefaea-deba-44ec-a357-4fb4ba5e7f6e" path="/var/lib/kubelet/pods/aefefaea-deba-44ec-a357-4fb4ba5e7f6e/volumes"
Oct 30 17:38:56.977643 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:38:56.977602    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:39:02.035897 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:02.035855    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33075c97-7mc6t"]
Oct 30 17:39:02.047176 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:02.047144    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33075c97-7mc6t"]
Oct 30 17:39:03.695546 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:03.695514    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35956888-3e58-4794-8613-bf44931f5129" path="/var/lib/kubelet/pods/35956888-3e58-4794-8613-bf44931f5129/volumes"
Oct 30 17:39:04.362513 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:04.362474    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:39:06.085737 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:06.085702    4433 scope.go:122] "RemoveContainer" containerID="5dd6b88c1ab651473c99b6fefa002f2afbfbf778cda5c447f75d13684c7202b2"
Oct 30 17:39:06.097096 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:06.097064    4433 scope.go:122] "RemoveContainer" containerID="be2aae9144715a73a9253392ac6ac738029d9a35b9f688d272bb8b5389660b98"
Oct 30 17:39:06.107972 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:06.107946    4433 scope.go:122] "RemoveContainer" containerID="0c579ac46245675fb05cf732740f8bcd8b1afa705d5e4b4d06dd0cf9d977bafb"
Oct 30 17:39:06.134067 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:06.134027    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" containerID="cri-o://906ad64932fa3b5bdec53a115d4bf5e51a83cb4e350d21ca105c30ef0260476c" gracePeriod=45
Oct 30 17:39:06.244721 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:06.244683    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": dial tcp 10.133.0.40:8000: connect: connection refused"
Oct 30 17:39:06.424795 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:06.424763    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 17:39:06.499169 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:06.492122    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c99f0236-6725-41a4-abd1-0bc0d9573910-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c99f0236-6725-41a4-abd1-0bc0d9573910-tmp-dir\") pod \"c99f0236-6725-41a4-abd1-0bc0d9573910\" (UID: \"c99f0236-6725-41a4-abd1-0bc0d9573910\") "
Oct 30 17:39:06.499169 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:06.492174    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c99f0236-6725-41a4-abd1-0bc0d9573910-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c99f0236-6725-41a4-abd1-0bc0d9573910-model-cache\") pod \"c99f0236-6725-41a4-abd1-0bc0d9573910\" (UID: \"c99f0236-6725-41a4-abd1-0bc0d9573910\") "
Oct 30 17:39:06.499169 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:06.492250    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c99f0236-6725-41a4-abd1-0bc0d9573910-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c99f0236-6725-41a4-abd1-0bc0d9573910-dshm\") pod \"c99f0236-6725-41a4-abd1-0bc0d9573910\" (UID: \"c99f0236-6725-41a4-abd1-0bc0d9573910\") "
Oct 30 17:39:06.499169 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:06.492299    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c99f0236-6725-41a4-abd1-0bc0d9573910-kube-api-access-pz9cf\" (UniqueName: \"kubernetes.io/projected/c99f0236-6725-41a4-abd1-0bc0d9573910-kube-api-access-pz9cf\") pod \"c99f0236-6725-41a4-abd1-0bc0d9573910\" (UID: \"c99f0236-6725-41a4-abd1-0bc0d9573910\") "
Oct 30 17:39:06.499169 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:06.492326    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c99f0236-6725-41a4-abd1-0bc0d9573910-home\" (UniqueName: \"kubernetes.io/empty-dir/c99f0236-6725-41a4-abd1-0bc0d9573910-home\") pod \"c99f0236-6725-41a4-abd1-0bc0d9573910\" (UID: \"c99f0236-6725-41a4-abd1-0bc0d9573910\") "
Oct 30 17:39:06.499169 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:06.492430    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c99f0236-6725-41a4-abd1-0bc0d9573910-tls-certs\" (UniqueName: \"kubernetes.io/secret/c99f0236-6725-41a4-abd1-0bc0d9573910-tls-certs\") pod \"c99f0236-6725-41a4-abd1-0bc0d9573910\" (UID: \"c99f0236-6725-41a4-abd1-0bc0d9573910\") "
Oct 30 17:39:06.499169 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:06.492469    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c99f0236-6725-41a4-abd1-0bc0d9573910-tmp-dir" pod "c99f0236-6725-41a4-abd1-0bc0d9573910" (UID: "c99f0236-6725-41a4-abd1-0bc0d9573910"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:39:06.499169 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:06.492687    4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c99f0236-6725-41a4-abd1-0bc0d9573910-tmp-dir\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:39:06.499169 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:06.493625    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c99f0236-6725-41a4-abd1-0bc0d9573910-model-cache" pod "c99f0236-6725-41a4-abd1-0bc0d9573910" (UID: "c99f0236-6725-41a4-abd1-0bc0d9573910"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:39:06.499169 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:06.494112    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c99f0236-6725-41a4-abd1-0bc0d9573910-home" pod "c99f0236-6725-41a4-abd1-0bc0d9573910" (UID: "c99f0236-6725-41a4-abd1-0bc0d9573910"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:39:06.499169 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:06.495269    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c99f0236-6725-41a4-abd1-0bc0d9573910-dshm" pod "c99f0236-6725-41a4-abd1-0bc0d9573910" (UID: "c99f0236-6725-41a4-abd1-0bc0d9573910"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:39:06.499169 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:06.495479    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c99f0236-6725-41a4-abd1-0bc0d9573910-kube-api-access-pz9cf" pod "c99f0236-6725-41a4-abd1-0bc0d9573910" (UID: "c99f0236-6725-41a4-abd1-0bc0d9573910"). InnerVolumeSpecName "kube-api-access-pz9cf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:39:06.499169 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:06.495503    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c99f0236-6725-41a4-abd1-0bc0d9573910-tls-certs" pod "c99f0236-6725-41a4-abd1-0bc0d9573910" (UID: "c99f0236-6725-41a4-abd1-0bc0d9573910"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:39:06.593743 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:06.593711    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz9cf\" (UniqueName: \"kubernetes.io/projected/c99f0236-6725-41a4-abd1-0bc0d9573910-kube-api-access-pz9cf\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:39:06.593743 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:06.593740    4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c99f0236-6725-41a4-abd1-0bc0d9573910-home\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:39:06.593958 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:06.593752    4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c99f0236-6725-41a4-abd1-0bc0d9573910-tls-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:39:06.593958 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:06.593761    4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c99f0236-6725-41a4-abd1-0bc0d9573910-model-cache\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:39:06.593958 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:06.593769    4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c99f0236-6725-41a4-abd1-0bc0d9573910-dshm\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:39:06.910279 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:06.910247    4433 generic.go:356] "Generic (PLEG): container finished" podID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerID="906ad64932fa3b5bdec53a115d4bf5e51a83cb4e350d21ca105c30ef0260476c" exitCode=0
Oct 30 17:39:06.910478 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:06.910291    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"
Oct 30 17:39:06.910478 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:06.910331    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" event={"ID":"c99f0236-6725-41a4-abd1-0bc0d9573910","Type":"ContainerDied","Data":"906ad64932fa3b5bdec53a115d4bf5e51a83cb4e350d21ca105c30ef0260476c"}
Oct 30 17:39:06.910478 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:06.910373    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z" event={"ID":"c99f0236-6725-41a4-abd1-0bc0d9573910","Type":"ContainerDied","Data":"e8948d5897b3b31987e2fef9f1794d70cb4b066f4e766d623a85f6acee5791c4"}
Oct 30 17:39:06.910478 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:06.910393    4433 scope.go:122] "RemoveContainer" containerID="906ad64932fa3b5bdec53a115d4bf5e51a83cb4e350d21ca105c30ef0260476c"
Oct 30 17:39:06.919931 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:06.919911    4433 scope.go:122] "RemoveContainer" containerID="906ad64932fa3b5bdec53a115d4bf5e51a83cb4e350d21ca105c30ef0260476c"
Oct 30 17:39:06.920220 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:39:06.920198    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"906ad64932fa3b5bdec53a115d4bf5e51a83cb4e350d21ca105c30ef0260476c\": container with ID starting with 906ad64932fa3b5bdec53a115d4bf5e51a83cb4e350d21ca105c30ef0260476c not found: ID does not exist" containerID="906ad64932fa3b5bdec53a115d4bf5e51a83cb4e350d21ca105c30ef0260476c"
Oct 30 17:39:06.920283 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:06.920229    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"906ad64932fa3b5bdec53a115d4bf5e51a83cb4e350d21ca105c30ef0260476c"} err="failed to get container status \"906ad64932fa3b5bdec53a115d4bf5e51a83cb4e350d21ca105c30ef0260476c\": rpc error: code = NotFound desc = could not find container \"906ad64932fa3b5bdec53a115d4bf5e51a83cb4e350d21ca105c30ef0260476c\": container with ID starting with 906ad64932fa3b5bdec53a115d4bf5e51a83cb4e350d21ca105c30ef0260476c not found: ID does not exist"
Oct 30 17:39:06.959383 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:06.959351    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"]
Oct 30 17:39:06.970440 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:06.970398    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-096d75/simulator-w0-096d75-kserve-5cf76ddfdd-c599z"]
Oct 30 17:39:07.694092 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:07.694059    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" path="/var/lib/kubelet/pods/c99f0236-6725-41a4-abd1-0bc0d9573910/volumes"
Oct 30 17:39:18.855617 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:39:18.855585    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:39:26.976100 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:39:26.976057    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:39:56.977576 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:39:56.977538    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:40:06.149830 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:06.149795    4433 scope.go:122] "RemoveContainer" containerID="ff52d06aece677ba41bd21974a28579ca10e6dbe4056967abb30e8de21b60f3e"
Oct 30 17:40:06.158949 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:06.158925    4433 scope.go:122] "RemoveContainer" containerID="4efbdf4e1551f7165b9fc06cc98d70cb25b8da019beca492355ad64c42591e69"
Oct 30 17:40:06.166983 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:06.166966    4433 scope.go:122] "RemoveContainer" containerID="f217cf4f0f265e1956b307ea0ac3937ab98cdac725754e158a879779853527fe"
Oct 30 17:40:06.174649 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:06.174622    4433 scope.go:122] "RemoveContainer" containerID="780ea53cd9efd134d85dd4c700479f441e5930a41885888456d58e225cc30d64"
Oct 30 17:40:06.182432 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:06.182392    4433 scope.go:122] "RemoveContainer" containerID="45856953bdd048ca79b2510fd0fbbd7381e72ecbbd5397d1a9d88e70d1ac158b"
Oct 30 17:40:11.855195 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:11.855162    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:40:26.979468 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:40:26.979428    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:40:26.987724 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:26.987687    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns"]
Oct 30 17:40:26.987996 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:26.987960    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns" podUID="73875344-e1af-4208-86a8-681a05925c66" containerName="maas-api" containerID="cri-o://97803310540c97573daf259b93926a3f8d15c2d8276a4be2a06dd69b3e6fdec2" gracePeriod=30
Oct 30 17:40:27.272478 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:27.272449    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns"
Oct 30 17:40:27.326846 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:27.326806    4433 generic.go:356] "Generic (PLEG): container finished" podID="73875344-e1af-4208-86a8-681a05925c66" containerID="97803310540c97573daf259b93926a3f8d15c2d8276a4be2a06dd69b3e6fdec2" exitCode=0
Oct 30 17:40:27.327033 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:27.326855    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns"
Oct 30 17:40:27.327033 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:27.326893    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns" event={"ID":"73875344-e1af-4208-86a8-681a05925c66","Type":"ContainerDied","Data":"97803310540c97573daf259b93926a3f8d15c2d8276a4be2a06dd69b3e6fdec2"}
Oct 30 17:40:27.327033 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:27.326955    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns" event={"ID":"73875344-e1af-4208-86a8-681a05925c66","Type":"ContainerDied","Data":"9364b82f83344ed1b60ba329b92f0f31245b333fdb5db394bb479dc9c558364c"}
Oct 30 17:40:27.327033 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:27.326976    4433 scope.go:122] "RemoveContainer" containerID="97803310540c97573daf259b93926a3f8d15c2d8276a4be2a06dd69b3e6fdec2"
Oct 30 17:40:27.340118 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:27.340091    4433 scope.go:122] "RemoveContainer" containerID="97803310540c97573daf259b93926a3f8d15c2d8276a4be2a06dd69b3e6fdec2"
Oct 30 17:40:27.340548 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:40:27.340519    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97803310540c97573daf259b93926a3f8d15c2d8276a4be2a06dd69b3e6fdec2\": container with ID starting with 97803310540c97573daf259b93926a3f8d15c2d8276a4be2a06dd69b3e6fdec2 not found: ID does not exist" containerID="97803310540c97573daf259b93926a3f8d15c2d8276a4be2a06dd69b3e6fdec2"
Oct 30 17:40:27.340672 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:27.340560    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97803310540c97573daf259b93926a3f8d15c2d8276a4be2a06dd69b3e6fdec2"} err="failed to get container status \"97803310540c97573daf259b93926a3f8d15c2d8276a4be2a06dd69b3e6fdec2\": rpc error: code = NotFound desc = could not find container \"97803310540c97573daf259b93926a3f8d15c2d8276a4be2a06dd69b3e6fdec2\": container with ID starting with 97803310540c97573daf259b93926a3f8d15c2d8276a4be2a06dd69b3e6fdec2 not found: ID does not exist"
Oct 30 17:40:27.417338 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:27.417300    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73875344-e1af-4208-86a8-681a05925c66-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73875344-e1af-4208-86a8-681a05925c66-openshift-service-ca\") pod \"73875344-e1af-4208-86a8-681a05925c66\" (UID: \"73875344-e1af-4208-86a8-681a05925c66\") "
Oct 30 17:40:27.417556 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:27.417451    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73875344-e1af-4208-86a8-681a05925c66-kube-api-access-vrldx\" (UniqueName: \"kubernetes.io/projected/73875344-e1af-4208-86a8-681a05925c66-kube-api-access-vrldx\") pod \"73875344-e1af-4208-86a8-681a05925c66\" (UID: \"73875344-e1af-4208-86a8-681a05925c66\") "
Oct 30 17:40:27.417556 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:27.417476    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73875344-e1af-4208-86a8-681a05925c66-metrics-certs\" (UniqueName: \"kubernetes.io/secret/73875344-e1af-4208-86a8-681a05925c66-metrics-certs\") pod \"73875344-e1af-4208-86a8-681a05925c66\" (UID: \"73875344-e1af-4208-86a8-681a05925c66\") "
Oct 30 17:40:27.417556 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:27.417503    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73875344-e1af-4208-86a8-681a05925c66-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73875344-e1af-4208-86a8-681a05925c66-maas-api-tls\") pod \"73875344-e1af-4208-86a8-681a05925c66\" (UID: \"73875344-e1af-4208-86a8-681a05925c66\") "
Oct 30 17:40:27.417741 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:27.417707    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73875344-e1af-4208-86a8-681a05925c66-openshift-service-ca" pod "73875344-e1af-4208-86a8-681a05925c66" (UID: "73875344-e1af-4208-86a8-681a05925c66"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:40:27.417973 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:27.417953    4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73875344-e1af-4208-86a8-681a05925c66-openshift-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:40:27.419684 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:27.419651    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73875344-e1af-4208-86a8-681a05925c66-metrics-certs" pod "73875344-e1af-4208-86a8-681a05925c66" (UID: "73875344-e1af-4208-86a8-681a05925c66"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:40:27.419801 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:27.419709    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73875344-e1af-4208-86a8-681a05925c66-kube-api-access-vrldx" pod "73875344-e1af-4208-86a8-681a05925c66" (UID: "73875344-e1af-4208-86a8-681a05925c66"). InnerVolumeSpecName "kube-api-access-vrldx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:40:27.419801 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:27.419716    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73875344-e1af-4208-86a8-681a05925c66-maas-api-tls" pod "73875344-e1af-4208-86a8-681a05925c66" (UID: "73875344-e1af-4208-86a8-681a05925c66"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:40:27.519214 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:27.519118    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrldx\" (UniqueName: \"kubernetes.io/projected/73875344-e1af-4208-86a8-681a05925c66-kube-api-access-vrldx\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:40:27.519214 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:27.519156    4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73875344-e1af-4208-86a8-681a05925c66-metrics-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:40:27.519214 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:27.519168    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73875344-e1af-4208-86a8-681a05925c66-maas-api-tls\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:40:27.671369 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:27.671329    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns"]
Oct 30 17:40:27.682540 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:27.682507    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b87f3c-68958b8f8-l96ns"]
Oct 30 17:40:27.695461 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:27.695427    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73875344-e1af-4208-86a8-681a05925c66" path="/var/lib/kubelet/pods/73875344-e1af-4208-86a8-681a05925c66/volumes"
Oct 30 17:40:31.572490 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:31.572448    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:40:35.878557 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:35.878521    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"]
Oct 30 17:40:35.879123 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:35.878755    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz" podUID="51cf9cfa-eed9-4472-ba73-d696ef7bbfcf" containerName="istio-proxy" containerID="cri-o://9f7e96e18abec969a58085659e4f318835ad3d96e24ca6f34274e76819bc4059" gracePeriod=30
Oct 30 17:40:41.135889 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.135860    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"
Oct 30 17:40:41.154619 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.154553    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-istio-envoy\") pod \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") "
Oct 30 17:40:41.154619 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.154587    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-workload-socket\") pod \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") "
Oct 30 17:40:41.154619 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.154607    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-istio-token\" (UniqueName: \"kubernetes.io/projected/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-istio-token\") pod \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") "
Oct 30 17:40:41.154878 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.154651    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-istio-podinfo\") pod \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") "
Oct 30 17:40:41.154878 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.154674    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-workload-certs\") pod \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") "
Oct 30 17:40:41.154878 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.154699    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-credential-socket\") pod \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") "
Oct 30 17:40:41.154878 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.154757    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-istiod-ca-cert\") pod \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") "
Oct 30 17:40:41.154878 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.154826    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-istio-data\") pod \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") "
Oct 30 17:40:41.154878 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.154856    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-workload-socket" pod "51cf9cfa-eed9-4472-ba73-d696ef7bbfcf" (UID: "51cf9cfa-eed9-4472-ba73-d696ef7bbfcf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:40:41.155204 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.154910    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-kube-api-access-4r5s2\" (UniqueName: \"kubernetes.io/projected/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-kube-api-access-4r5s2\") pod \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\" (UID: \"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf\") "
Oct 30 17:40:41.155204 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.155120    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-credential-socket" pod "51cf9cfa-eed9-4472-ba73-d696ef7bbfcf" (UID: "51cf9cfa-eed9-4472-ba73-d696ef7bbfcf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:40:41.155204 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.155151    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-workload-certs" pod "51cf9cfa-eed9-4472-ba73-d696ef7bbfcf" (UID: "51cf9cfa-eed9-4472-ba73-d696ef7bbfcf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:40:41.155326 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.155235    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-istiod-ca-cert" pod "51cf9cfa-eed9-4472-ba73-d696ef7bbfcf" (UID: "51cf9cfa-eed9-4472-ba73-d696ef7bbfcf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:40:41.155399 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.155378    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:40:41.155496 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.155430    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:40:41.155496 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.155446    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:40:41.155496 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.155462    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:40:41.155675 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.155652    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-istio-data" pod "51cf9cfa-eed9-4472-ba73-d696ef7bbfcf" (UID: "51cf9cfa-eed9-4472-ba73-d696ef7bbfcf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:40:41.156925 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.156887    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-istio-envoy" pod "51cf9cfa-eed9-4472-ba73-d696ef7bbfcf" (UID: "51cf9cfa-eed9-4472-ba73-d696ef7bbfcf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:40:41.157048 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.156930    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-istio-podinfo" pod "51cf9cfa-eed9-4472-ba73-d696ef7bbfcf" (UID: "51cf9cfa-eed9-4472-ba73-d696ef7bbfcf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:40:41.157048 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.156986    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-istio-token" pod "51cf9cfa-eed9-4472-ba73-d696ef7bbfcf" (UID: "51cf9cfa-eed9-4472-ba73-d696ef7bbfcf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:40:41.157207 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.157191    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-kube-api-access-4r5s2" pod "51cf9cfa-eed9-4472-ba73-d696ef7bbfcf" (UID: "51cf9cfa-eed9-4472-ba73-d696ef7bbfcf"). InnerVolumeSpecName "kube-api-access-4r5s2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:40:41.256647 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.256612    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:40:41.256647 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.256640    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r5s2\" (UniqueName: \"kubernetes.io/projected/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-kube-api-access-4r5s2\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:40:41.256647 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.256650    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:40:41.256647 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.256658    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:40:41.256921 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.256667    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:40:41.399672 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.399641    4433 generic.go:356] "Generic (PLEG): container finished" podID="51cf9cfa-eed9-4472-ba73-d696ef7bbfcf" containerID="9f7e96e18abec969a58085659e4f318835ad3d96e24ca6f34274e76819bc4059" exitCode=0
Oct 30 17:40:41.399857 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.399690    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"
Oct 30 17:40:41.399857 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.399728    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz" event={"ID":"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf","Type":"ContainerDied","Data":"9f7e96e18abec969a58085659e4f318835ad3d96e24ca6f34274e76819bc4059"}
Oct 30 17:40:41.399857 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.399793    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz" event={"ID":"51cf9cfa-eed9-4472-ba73-d696ef7bbfcf","Type":"ContainerDied","Data":"c360d7bcf148e344d96287397693b6d7b27aed10262d9da1cddcf555792fba7a"}
Oct 30 17:40:41.399857 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.399823    4433 scope.go:122] "RemoveContainer" containerID="9f7e96e18abec969a58085659e4f318835ad3d96e24ca6f34274e76819bc4059"
Oct 30 17:40:41.411363 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.411338    4433 scope.go:122] "RemoveContainer" containerID="9f7e96e18abec969a58085659e4f318835ad3d96e24ca6f34274e76819bc4059"
Oct 30 17:40:41.411701 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:40:41.411673    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f7e96e18abec969a58085659e4f318835ad3d96e24ca6f34274e76819bc4059\": container with ID starting with 9f7e96e18abec969a58085659e4f318835ad3d96e24ca6f34274e76819bc4059 not found: ID does not exist" containerID="9f7e96e18abec969a58085659e4f318835ad3d96e24ca6f34274e76819bc4059"
Oct 30 17:40:41.411757 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.411714    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f7e96e18abec969a58085659e4f318835ad3d96e24ca6f34274e76819bc4059"} err="failed to get container status \"9f7e96e18abec969a58085659e4f318835ad3d96e24ca6f34274e76819bc4059\": rpc error: code = NotFound desc = could not find container \"9f7e96e18abec969a58085659e4f318835ad3d96e24ca6f34274e76819bc4059\": container with ID starting with 9f7e96e18abec969a58085659e4f318835ad3d96e24ca6f34274e76819bc4059 not found: ID does not exist"
Oct 30 17:40:41.453984 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.453942    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"]
Oct 30 17:40:41.459114 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.459086    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-8vpbz"]
Oct 30 17:40:41.694349 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:41.694309    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51cf9cfa-eed9-4472-ba73-d696ef7bbfcf" path="/var/lib/kubelet/pods/51cf9cfa-eed9-4472-ba73-d696ef7bbfcf/volumes"
Oct 30 17:40:50.772098 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:40:50.772059    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:40:56.978117 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:40:56.978074    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:41:01.694943 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:01.694900    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"]
Oct 30 17:41:01.705943 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:01.705913    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"]
Oct 30 17:41:16.716375 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:16.716315    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" podUID="458eba2e-eb03-452a-bcf4-a5e91da83acf" containerName="main" containerID="cri-o://8f562d015630ec70355311e02425fafc2d096fd7dc16a1e5ac07b7dc5c1100b7" gracePeriod=45
Oct 30 17:41:16.728924 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:16.728876    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main" containerID="cri-o://a2dea290264a109e51f06f3e1e92c061ed6a1e4619e214b4b9269d75ff04e062" gracePeriod=45
Oct 30 17:41:16.835508 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:16.835465    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" podUID="458eba2e-eb03-452a-bcf4-a5e91da83acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": dial tcp 10.133.0.65:8000: connect: connection refused"
Oct 30 17:41:16.986148 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:16.986119    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:41:17.014127 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.014100    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:41:17.016332 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.016311    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb78a5ca-3157-4707-a75f-f38748185b81-tls-certs\" (UniqueName: \"kubernetes.io/secret/bb78a5ca-3157-4707-a75f-f38748185b81-tls-certs\") pod \"bb78a5ca-3157-4707-a75f-f38748185b81\" (UID: \"bb78a5ca-3157-4707-a75f-f38748185b81\") "
Oct 30 17:41:17.016419 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.016345    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/458eba2e-eb03-452a-bcf4-a5e91da83acf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/458eba2e-eb03-452a-bcf4-a5e91da83acf-model-cache\") pod \"458eba2e-eb03-452a-bcf4-a5e91da83acf\" (UID: \"458eba2e-eb03-452a-bcf4-a5e91da83acf\") "
Oct 30 17:41:17.016419 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.016370    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb78a5ca-3157-4707-a75f-f38748185b81-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb78a5ca-3157-4707-a75f-f38748185b81-model-cache\") pod \"bb78a5ca-3157-4707-a75f-f38748185b81\" (UID: \"bb78a5ca-3157-4707-a75f-f38748185b81\") "
Oct 30 17:41:17.016501 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.016396    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb78a5ca-3157-4707-a75f-f38748185b81-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb78a5ca-3157-4707-a75f-f38748185b81-dshm\") pod \"bb78a5ca-3157-4707-a75f-f38748185b81\" (UID: \"bb78a5ca-3157-4707-a75f-f38748185b81\") "
Oct 30 17:41:17.016501 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.016455    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/458eba2e-eb03-452a-bcf4-a5e91da83acf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/458eba2e-eb03-452a-bcf4-a5e91da83acf-dshm\") pod \"458eba2e-eb03-452a-bcf4-a5e91da83acf\" (UID: \"458eba2e-eb03-452a-bcf4-a5e91da83acf\") "
Oct 30 17:41:17.016501 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.016487    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/458eba2e-eb03-452a-bcf4-a5e91da83acf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/458eba2e-eb03-452a-bcf4-a5e91da83acf-tmp-dir\") pod \"458eba2e-eb03-452a-bcf4-a5e91da83acf\" (UID: \"458eba2e-eb03-452a-bcf4-a5e91da83acf\") "
Oct 30 17:41:17.016670 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.016516    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb78a5ca-3157-4707-a75f-f38748185b81-home\" (UniqueName: \"kubernetes.io/empty-dir/bb78a5ca-3157-4707-a75f-f38748185b81-home\") pod \"bb78a5ca-3157-4707-a75f-f38748185b81\" (UID: \"bb78a5ca-3157-4707-a75f-f38748185b81\") "
Oct 30 17:41:17.016670 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.016545    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/458eba2e-eb03-452a-bcf4-a5e91da83acf-home\" (UniqueName: \"kubernetes.io/empty-dir/458eba2e-eb03-452a-bcf4-a5e91da83acf-home\") pod \"458eba2e-eb03-452a-bcf4-a5e91da83acf\" (UID: \"458eba2e-eb03-452a-bcf4-a5e91da83acf\") "
Oct 30 17:41:17.016670 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.016598    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb78a5ca-3157-4707-a75f-f38748185b81-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb78a5ca-3157-4707-a75f-f38748185b81-tmp-dir\") pod \"bb78a5ca-3157-4707-a75f-f38748185b81\" (UID: \"bb78a5ca-3157-4707-a75f-f38748185b81\") "
Oct 30 17:41:17.016670 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.016631    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb78a5ca-3157-4707-a75f-f38748185b81-kube-api-access-7g86z\" (UniqueName: \"kubernetes.io/projected/bb78a5ca-3157-4707-a75f-f38748185b81-kube-api-access-7g86z\") pod \"bb78a5ca-3157-4707-a75f-f38748185b81\" (UID: \"bb78a5ca-3157-4707-a75f-f38748185b81\") "
Oct 30 17:41:17.016670 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.016642    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/458eba2e-eb03-452a-bcf4-a5e91da83acf-model-cache" pod "458eba2e-eb03-452a-bcf4-a5e91da83acf" (UID: "458eba2e-eb03-452a-bcf4-a5e91da83acf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:17.016670 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.016664    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/458eba2e-eb03-452a-bcf4-a5e91da83acf-kube-api-access-88hvm\" (UniqueName: \"kubernetes.io/projected/458eba2e-eb03-452a-bcf4-a5e91da83acf-kube-api-access-88hvm\") pod \"458eba2e-eb03-452a-bcf4-a5e91da83acf\" (UID: \"458eba2e-eb03-452a-bcf4-a5e91da83acf\") "
Oct 30 17:41:17.016997 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.016728    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/458eba2e-eb03-452a-bcf4-a5e91da83acf-tls-certs\" (UniqueName: \"kubernetes.io/secret/458eba2e-eb03-452a-bcf4-a5e91da83acf-tls-certs\") pod \"458eba2e-eb03-452a-bcf4-a5e91da83acf\" (UID: \"458eba2e-eb03-452a-bcf4-a5e91da83acf\") "
Oct 30 17:41:17.016997 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.016641    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb78a5ca-3157-4707-a75f-f38748185b81-model-cache" pod "bb78a5ca-3157-4707-a75f-f38748185b81" (UID: "bb78a5ca-3157-4707-a75f-f38748185b81"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:17.016997 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.016871    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/458eba2e-eb03-452a-bcf4-a5e91da83acf-home" pod "458eba2e-eb03-452a-bcf4-a5e91da83acf" (UID: "458eba2e-eb03-452a-bcf4-a5e91da83acf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:17.016997 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.016833    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb78a5ca-3157-4707-a75f-f38748185b81-tmp-dir" pod "bb78a5ca-3157-4707-a75f-f38748185b81" (UID: "bb78a5ca-3157-4707-a75f-f38748185b81"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:17.016997 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.016826    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/458eba2e-eb03-452a-bcf4-a5e91da83acf-tmp-dir" pod "458eba2e-eb03-452a-bcf4-a5e91da83acf" (UID: "458eba2e-eb03-452a-bcf4-a5e91da83acf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:17.017289 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.017076    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb78a5ca-3157-4707-a75f-f38748185b81-home" pod "bb78a5ca-3157-4707-a75f-f38748185b81" (UID: "bb78a5ca-3157-4707-a75f-f38748185b81"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:17.017289 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.017080    4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/458eba2e-eb03-452a-bcf4-a5e91da83acf-model-cache\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:41:17.017289 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.017105    4433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb78a5ca-3157-4707-a75f-f38748185b81-model-cache\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:41:17.017289 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.017114    4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/458eba2e-eb03-452a-bcf4-a5e91da83acf-tmp-dir\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:41:17.017289 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.017122    4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/458eba2e-eb03-452a-bcf4-a5e91da83acf-home\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:41:17.017289 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.017130    4433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb78a5ca-3157-4707-a75f-f38748185b81-tmp-dir\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:41:17.019111 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.019085    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/458eba2e-eb03-452a-bcf4-a5e91da83acf-tls-certs" pod "458eba2e-eb03-452a-bcf4-a5e91da83acf" (UID: "458eba2e-eb03-452a-bcf4-a5e91da83acf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:41:17.019223 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.019111    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb78a5ca-3157-4707-a75f-f38748185b81-kube-api-access-7g86z" pod "bb78a5ca-3157-4707-a75f-f38748185b81" (UID: "bb78a5ca-3157-4707-a75f-f38748185b81"). InnerVolumeSpecName "kube-api-access-7g86z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:41:17.019223 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.019132    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb78a5ca-3157-4707-a75f-f38748185b81-dshm" pod "bb78a5ca-3157-4707-a75f-f38748185b81" (UID: "bb78a5ca-3157-4707-a75f-f38748185b81"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:17.019223 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.019142    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb78a5ca-3157-4707-a75f-f38748185b81-tls-certs" pod "bb78a5ca-3157-4707-a75f-f38748185b81" (UID: "bb78a5ca-3157-4707-a75f-f38748185b81"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:41:17.019357 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.019274    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/458eba2e-eb03-452a-bcf4-a5e91da83acf-kube-api-access-88hvm" pod "458eba2e-eb03-452a-bcf4-a5e91da83acf" (UID: "458eba2e-eb03-452a-bcf4-a5e91da83acf"). InnerVolumeSpecName "kube-api-access-88hvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:41:17.019396 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.019366    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/458eba2e-eb03-452a-bcf4-a5e91da83acf-dshm" pod "458eba2e-eb03-452a-bcf4-a5e91da83acf" (UID: "458eba2e-eb03-452a-bcf4-a5e91da83acf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:17.118296 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.118259    4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb78a5ca-3157-4707-a75f-f38748185b81-tls-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:41:17.118296 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.118293    4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb78a5ca-3157-4707-a75f-f38748185b81-dshm\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:41:17.118296 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.118302    4433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/458eba2e-eb03-452a-bcf4-a5e91da83acf-dshm\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:41:17.118638 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.118309    4433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb78a5ca-3157-4707-a75f-f38748185b81-home\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:41:17.118638 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.118318    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g86z\" (UniqueName: \"kubernetes.io/projected/bb78a5ca-3157-4707-a75f-f38748185b81-kube-api-access-7g86z\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:41:17.118638 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.118327    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88hvm\" (UniqueName: \"kubernetes.io/projected/458eba2e-eb03-452a-bcf4-a5e91da83acf-kube-api-access-88hvm\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:41:17.118638 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.118336    4433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/458eba2e-eb03-452a-bcf4-a5e91da83acf-tls-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:41:17.580752 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.580714    4433 generic.go:356] "Generic (PLEG): container finished" podID="bb78a5ca-3157-4707-a75f-f38748185b81" containerID="a2dea290264a109e51f06f3e1e92c061ed6a1e4619e214b4b9269d75ff04e062" exitCode=0
Oct 30 17:41:17.580961 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.580777    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"
Oct 30 17:41:17.580961 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.580796    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" event={"ID":"bb78a5ca-3157-4707-a75f-f38748185b81","Type":"ContainerDied","Data":"a2dea290264a109e51f06f3e1e92c061ed6a1e4619e214b4b9269d75ff04e062"}
Oct 30 17:41:17.580961 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.580849    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg" event={"ID":"bb78a5ca-3157-4707-a75f-f38748185b81","Type":"ContainerDied","Data":"4ea30057cf180ff68e72643094aef8b1125c3a21a98bf9b4df51f257e5e1a131"}
Oct 30 17:41:17.580961 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.580873    4433 scope.go:122] "RemoveContainer" containerID="a2dea290264a109e51f06f3e1e92c061ed6a1e4619e214b4b9269d75ff04e062"
Oct 30 17:41:17.582190 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.582168    4433 generic.go:356] "Generic (PLEG): container finished" podID="458eba2e-eb03-452a-bcf4-a5e91da83acf" containerID="8f562d015630ec70355311e02425fafc2d096fd7dc16a1e5ac07b7dc5c1100b7" exitCode=0
Oct 30 17:41:17.582308 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.582226    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"
Oct 30 17:41:17.582308 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.582268    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" event={"ID":"458eba2e-eb03-452a-bcf4-a5e91da83acf","Type":"ContainerDied","Data":"8f562d015630ec70355311e02425fafc2d096fd7dc16a1e5ac07b7dc5c1100b7"}
Oct 30 17:41:17.582437 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.582311    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj" event={"ID":"458eba2e-eb03-452a-bcf4-a5e91da83acf","Type":"ContainerDied","Data":"6327cfaa38a16aadd30c0e8b707596dc505806df73406f4b6762baf427833469"}
Oct 30 17:41:17.590749 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.590727    4433 scope.go:122] "RemoveContainer" containerID="a2dea290264a109e51f06f3e1e92c061ed6a1e4619e214b4b9269d75ff04e062"
Oct 30 17:41:17.591221 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:41:17.591196    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2dea290264a109e51f06f3e1e92c061ed6a1e4619e214b4b9269d75ff04e062\": container with ID starting with a2dea290264a109e51f06f3e1e92c061ed6a1e4619e214b4b9269d75ff04e062 not found: ID does not exist" containerID="a2dea290264a109e51f06f3e1e92c061ed6a1e4619e214b4b9269d75ff04e062"
Oct 30 17:41:17.591331 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.591234    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2dea290264a109e51f06f3e1e92c061ed6a1e4619e214b4b9269d75ff04e062"} err="failed to get container status \"a2dea290264a109e51f06f3e1e92c061ed6a1e4619e214b4b9269d75ff04e062\": rpc error: code = NotFound desc = could not find container \"a2dea290264a109e51f06f3e1e92c061ed6a1e4619e214b4b9269d75ff04e062\": container with ID starting with a2dea290264a109e51f06f3e1e92c061ed6a1e4619e214b4b9269d75ff04e062 not found: ID does not exist"
Oct 30 17:41:17.591331 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.591259    4433 scope.go:122] "RemoveContainer" containerID="8f562d015630ec70355311e02425fafc2d096fd7dc16a1e5ac07b7dc5c1100b7"
Oct 30 17:41:17.599949 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.599922    4433 scope.go:122] "RemoveContainer" containerID="8f562d015630ec70355311e02425fafc2d096fd7dc16a1e5ac07b7dc5c1100b7"
Oct 30 17:41:17.600288 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:41:17.600268    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f562d015630ec70355311e02425fafc2d096fd7dc16a1e5ac07b7dc5c1100b7\": container with ID starting with 8f562d015630ec70355311e02425fafc2d096fd7dc16a1e5ac07b7dc5c1100b7 not found: ID does not exist" containerID="8f562d015630ec70355311e02425fafc2d096fd7dc16a1e5ac07b7dc5c1100b7"
Oct 30 17:41:17.600343 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.600299    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f562d015630ec70355311e02425fafc2d096fd7dc16a1e5ac07b7dc5c1100b7"} err="failed to get container status \"8f562d015630ec70355311e02425fafc2d096fd7dc16a1e5ac07b7dc5c1100b7\": rpc error: code = NotFound desc = could not find container \"8f562d015630ec70355311e02425fafc2d096fd7dc16a1e5ac07b7dc5c1100b7\": container with ID starting with 8f562d015630ec70355311e02425fafc2d096fd7dc16a1e5ac07b7dc5c1100b7 not found: ID does not exist"
Oct 30 17:41:17.631653 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.631611    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"]
Oct 30 17:41:17.641849 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.641806    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-d384dc/distinct-2-w6-d384dc-kserve-5b89788bbd-z5vbj"]
Oct 30 17:41:17.669839 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.669798    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"]
Oct 30 17:41:17.674057 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.674026    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-d384dc/distinct-w6-d384dc-kserve-88966dd47-6skpg"]
Oct 30 17:41:17.694315 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.694269    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="458eba2e-eb03-452a-bcf4-a5e91da83acf" path="/var/lib/kubelet/pods/458eba2e-eb03-452a-bcf4-a5e91da83acf/volumes"
Oct 30 17:41:17.694823 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:17.694799    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" path="/var/lib/kubelet/pods/bb78a5ca-3157-4707-a75f-f38748185b81/volumes"
Oct 30 17:41:22.445275 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:22.445235    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:41:24.765158 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:24.765122    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9"]
Oct 30 17:41:24.765652 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:24.765338    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9" podUID="186d5a68-3c34-4953-8e2b-66a1278b9dd7" containerName="maas-api" containerID="cri-o://c63cdc52fc4ae95864530e3e27cbd31a0c8fc67fc8079f78eb0c1c8db89e4fa9" gracePeriod=30
Oct 30 17:41:25.024193 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:25.024123    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9"
Oct 30 17:41:25.091191 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:25.091152    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/186d5a68-3c34-4953-8e2b-66a1278b9dd7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/186d5a68-3c34-4953-8e2b-66a1278b9dd7-metrics-certs\") pod \"186d5a68-3c34-4953-8e2b-66a1278b9dd7\" (UID: \"186d5a68-3c34-4953-8e2b-66a1278b9dd7\") "
Oct 30 17:41:25.091352 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:25.091202    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/186d5a68-3c34-4953-8e2b-66a1278b9dd7-kube-api-access-wjvrm\" (UniqueName: \"kubernetes.io/projected/186d5a68-3c34-4953-8e2b-66a1278b9dd7-kube-api-access-wjvrm\") pod \"186d5a68-3c34-4953-8e2b-66a1278b9dd7\" (UID: \"186d5a68-3c34-4953-8e2b-66a1278b9dd7\") "
Oct 30 17:41:25.091352 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:25.091256    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/186d5a68-3c34-4953-8e2b-66a1278b9dd7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/186d5a68-3c34-4953-8e2b-66a1278b9dd7-maas-api-tls\") pod \"186d5a68-3c34-4953-8e2b-66a1278b9dd7\" (UID: \"186d5a68-3c34-4953-8e2b-66a1278b9dd7\") "
Oct 30 17:41:25.091352 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:25.091275    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/186d5a68-3c34-4953-8e2b-66a1278b9dd7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/186d5a68-3c34-4953-8e2b-66a1278b9dd7-openshift-service-ca\") pod \"186d5a68-3c34-4953-8e2b-66a1278b9dd7\" (UID: \"186d5a68-3c34-4953-8e2b-66a1278b9dd7\") "
Oct 30 17:41:25.091796 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:25.091760    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/186d5a68-3c34-4953-8e2b-66a1278b9dd7-openshift-service-ca" pod "186d5a68-3c34-4953-8e2b-66a1278b9dd7" (UID: "186d5a68-3c34-4953-8e2b-66a1278b9dd7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:41:25.093564 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:25.093536    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/186d5a68-3c34-4953-8e2b-66a1278b9dd7-metrics-certs" pod "186d5a68-3c34-4953-8e2b-66a1278b9dd7" (UID: "186d5a68-3c34-4953-8e2b-66a1278b9dd7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:41:25.093990 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:25.093962    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/186d5a68-3c34-4953-8e2b-66a1278b9dd7-maas-api-tls" pod "186d5a68-3c34-4953-8e2b-66a1278b9dd7" (UID: "186d5a68-3c34-4953-8e2b-66a1278b9dd7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:41:25.094066 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:25.094009    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/186d5a68-3c34-4953-8e2b-66a1278b9dd7-kube-api-access-wjvrm" pod "186d5a68-3c34-4953-8e2b-66a1278b9dd7" (UID: "186d5a68-3c34-4953-8e2b-66a1278b9dd7"). InnerVolumeSpecName "kube-api-access-wjvrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:41:25.193046 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:25.192985    4433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/186d5a68-3c34-4953-8e2b-66a1278b9dd7-metrics-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:41:25.193046 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:25.193032    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjvrm\" (UniqueName: \"kubernetes.io/projected/186d5a68-3c34-4953-8e2b-66a1278b9dd7-kube-api-access-wjvrm\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:41:25.193046 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:25.193048    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/186d5a68-3c34-4953-8e2b-66a1278b9dd7-maas-api-tls\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:41:25.193441 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:25.193062    4433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/186d5a68-3c34-4953-8e2b-66a1278b9dd7-openshift-service-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:41:25.628821 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:25.628778    4433 generic.go:356] "Generic (PLEG): container finished" podID="186d5a68-3c34-4953-8e2b-66a1278b9dd7" containerID="c63cdc52fc4ae95864530e3e27cbd31a0c8fc67fc8079f78eb0c1c8db89e4fa9" exitCode=0
Oct 30 17:41:25.629009 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:25.628847    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9"
Oct 30 17:41:25.629009 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:25.628860    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9" event={"ID":"186d5a68-3c34-4953-8e2b-66a1278b9dd7","Type":"ContainerDied","Data":"c63cdc52fc4ae95864530e3e27cbd31a0c8fc67fc8079f78eb0c1c8db89e4fa9"}
Oct 30 17:41:25.629009 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:25.628950    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9" event={"ID":"186d5a68-3c34-4953-8e2b-66a1278b9dd7","Type":"ContainerDied","Data":"17c9830c25f4150f83c599a0e116be6973c9d9febf898e0fe2152292060a01e0"}
Oct 30 17:41:25.629009 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:25.628975    4433 scope.go:122] "RemoveContainer" containerID="c63cdc52fc4ae95864530e3e27cbd31a0c8fc67fc8079f78eb0c1c8db89e4fa9"
Oct 30 17:41:25.638949 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:25.638923    4433 scope.go:122] "RemoveContainer" containerID="c63cdc52fc4ae95864530e3e27cbd31a0c8fc67fc8079f78eb0c1c8db89e4fa9"
Oct 30 17:41:25.639240 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:41:25.639219    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c63cdc52fc4ae95864530e3e27cbd31a0c8fc67fc8079f78eb0c1c8db89e4fa9\": container with ID starting with c63cdc52fc4ae95864530e3e27cbd31a0c8fc67fc8079f78eb0c1c8db89e4fa9 not found: ID does not exist" containerID="c63cdc52fc4ae95864530e3e27cbd31a0c8fc67fc8079f78eb0c1c8db89e4fa9"
Oct 30 17:41:25.639295 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:25.639252    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c63cdc52fc4ae95864530e3e27cbd31a0c8fc67fc8079f78eb0c1c8db89e4fa9"} err="failed to get container status \"c63cdc52fc4ae95864530e3e27cbd31a0c8fc67fc8079f78eb0c1c8db89e4fa9\": rpc error: code = NotFound desc = could not find container \"c63cdc52fc4ae95864530e3e27cbd31a0c8fc67fc8079f78eb0c1c8db89e4fa9\": container with ID starting with c63cdc52fc4ae95864530e3e27cbd31a0c8fc67fc8079f78eb0c1c8db89e4fa9 not found: ID does not exist"
Oct 30 17:41:25.678339 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:25.678302    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9"]
Oct 30 17:41:25.694524 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:25.694496    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-d384dc-5cd4847498-lfqg9"]
Oct 30 17:41:26.979458 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:41:26.979423    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:41:27.693332 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:27.693284    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="186d5a68-3c34-4953-8e2b-66a1278b9dd7" path="/var/lib/kubelet/pods/186d5a68-3c34-4953-8e2b-66a1278b9dd7/volumes"
Oct 30 17:41:31.357832 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:31.357783    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:41:32.033672 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:32.033633    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-755a1b1c-vsptj"]
Oct 30 17:41:32.045037 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:32.045007    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-755a1b1c-vsptj"]
Oct 30 17:41:33.349329 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:33.349287    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"]
Oct 30 17:41:33.350115 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:33.350076    4433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6" podUID="c690570f-1cba-450b-bcf6-0549e1ded4b7" containerName="istio-proxy" containerID="cri-o://f52b5c2fa4ff4776c5fa09f942903fc5f3104f65ab5fc44add042f79c1160ea6" gracePeriod=30
Oct 30 17:41:33.695312 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:33.695273    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8ae8154-7aa8-45be-80f8-3b53bd2a6842" path="/var/lib/kubelet/pods/b8ae8154-7aa8-45be-80f8-3b53bd2a6842/volumes"
Oct 30 17:41:38.630632 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:38.630564    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"
Oct 30 17:41:38.696375 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:38.696344    4433 generic.go:356] "Generic (PLEG): container finished" podID="c690570f-1cba-450b-bcf6-0549e1ded4b7" containerID="f52b5c2fa4ff4776c5fa09f942903fc5f3104f65ab5fc44add042f79c1160ea6" exitCode=0
Oct 30 17:41:38.696597 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:38.696393    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"
Oct 30 17:41:38.696597 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:38.696457    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6" event={"ID":"c690570f-1cba-450b-bcf6-0549e1ded4b7","Type":"ContainerDied","Data":"f52b5c2fa4ff4776c5fa09f942903fc5f3104f65ab5fc44add042f79c1160ea6"}
Oct 30 17:41:38.696597 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:38.696529    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6" event={"ID":"c690570f-1cba-450b-bcf6-0549e1ded4b7","Type":"ContainerDied","Data":"0113dccd6661fc3cc4cf2bdd4dab369ae270075c5ee9a8680f764448efc62967"}
Oct 30 17:41:38.696597 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:38.696562    4433 scope.go:122] "RemoveContainer" containerID="f52b5c2fa4ff4776c5fa09f942903fc5f3104f65ab5fc44add042f79c1160ea6"
Oct 30 17:41:38.705944 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:38.705919    4433 scope.go:122] "RemoveContainer" containerID="f52b5c2fa4ff4776c5fa09f942903fc5f3104f65ab5fc44add042f79c1160ea6"
Oct 30 17:41:38.706263 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:41:38.706239    4433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f52b5c2fa4ff4776c5fa09f942903fc5f3104f65ab5fc44add042f79c1160ea6\": container with ID starting with f52b5c2fa4ff4776c5fa09f942903fc5f3104f65ab5fc44add042f79c1160ea6 not found: ID does not exist" containerID="f52b5c2fa4ff4776c5fa09f942903fc5f3104f65ab5fc44add042f79c1160ea6"
Oct 30 17:41:38.706320 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:38.706272    4433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f52b5c2fa4ff4776c5fa09f942903fc5f3104f65ab5fc44add042f79c1160ea6"} err="failed to get container status \"f52b5c2fa4ff4776c5fa09f942903fc5f3104f65ab5fc44add042f79c1160ea6\": rpc error: code = NotFound desc = could not find container \"f52b5c2fa4ff4776c5fa09f942903fc5f3104f65ab5fc44add042f79c1160ea6\": container with ID starting with f52b5c2fa4ff4776c5fa09f942903fc5f3104f65ab5fc44add042f79c1160ea6 not found: ID does not exist"
Oct 30 17:41:38.724422 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:38.724381    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c690570f-1cba-450b-bcf6-0549e1ded4b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c690570f-1cba-450b-bcf6-0549e1ded4b7-istio-podinfo\") pod \"c690570f-1cba-450b-bcf6-0549e1ded4b7\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") "
Oct 30 17:41:38.724608 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:38.724458    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c690570f-1cba-450b-bcf6-0549e1ded4b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c690570f-1cba-450b-bcf6-0549e1ded4b7-istiod-ca-cert\") pod \"c690570f-1cba-450b-bcf6-0549e1ded4b7\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") "
Oct 30 17:41:38.724608 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:38.724489    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c690570f-1cba-450b-bcf6-0549e1ded4b7-istio-token\" (UniqueName: \"kubernetes.io/projected/c690570f-1cba-450b-bcf6-0549e1ded4b7-istio-token\") pod \"c690570f-1cba-450b-bcf6-0549e1ded4b7\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") "
Oct 30 17:41:38.724608 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:38.724515    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c690570f-1cba-450b-bcf6-0549e1ded4b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c690570f-1cba-450b-bcf6-0549e1ded4b7-workload-socket\") pod \"c690570f-1cba-450b-bcf6-0549e1ded4b7\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") "
Oct 30 17:41:38.724608 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:38.724557    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c690570f-1cba-450b-bcf6-0549e1ded4b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c690570f-1cba-450b-bcf6-0549e1ded4b7-credential-socket\") pod \"c690570f-1cba-450b-bcf6-0549e1ded4b7\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") "
Oct 30 17:41:38.724856 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:38.724683    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c690570f-1cba-450b-bcf6-0549e1ded4b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c690570f-1cba-450b-bcf6-0549e1ded4b7-istio-data\") pod \"c690570f-1cba-450b-bcf6-0549e1ded4b7\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") "
Oct 30 17:41:38.724856 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:38.724724    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c690570f-1cba-450b-bcf6-0549e1ded4b7-kube-api-access-v7nxp\" (UniqueName: \"kubernetes.io/projected/c690570f-1cba-450b-bcf6-0549e1ded4b7-kube-api-access-v7nxp\") pod \"c690570f-1cba-450b-bcf6-0549e1ded4b7\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") "
Oct 30 17:41:38.724856 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:38.724766    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c690570f-1cba-450b-bcf6-0549e1ded4b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c690570f-1cba-450b-bcf6-0549e1ded4b7-workload-certs\") pod \"c690570f-1cba-450b-bcf6-0549e1ded4b7\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") "
Oct 30 17:41:38.724856 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:38.724810    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c690570f-1cba-450b-bcf6-0549e1ded4b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c690570f-1cba-450b-bcf6-0549e1ded4b7-istio-envoy\") pod \"c690570f-1cba-450b-bcf6-0549e1ded4b7\" (UID: \"c690570f-1cba-450b-bcf6-0549e1ded4b7\") "
Oct 30 17:41:38.725081 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:38.725006    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c690570f-1cba-450b-bcf6-0549e1ded4b7-credential-socket" pod "c690570f-1cba-450b-bcf6-0549e1ded4b7" (UID: "c690570f-1cba-450b-bcf6-0549e1ded4b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:38.725192 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:38.725167    4433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c690570f-1cba-450b-bcf6-0549e1ded4b7-credential-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:41:38.725561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:38.725523    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c690570f-1cba-450b-bcf6-0549e1ded4b7-istio-data" pod "c690570f-1cba-450b-bcf6-0549e1ded4b7" (UID: "c690570f-1cba-450b-bcf6-0549e1ded4b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:38.725561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:38.725540    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c690570f-1cba-450b-bcf6-0549e1ded4b7-workload-certs" pod "c690570f-1cba-450b-bcf6-0549e1ded4b7" (UID: "c690570f-1cba-450b-bcf6-0549e1ded4b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:38.725749 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:38.725573    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c690570f-1cba-450b-bcf6-0549e1ded4b7-workload-socket" pod "c690570f-1cba-450b-bcf6-0549e1ded4b7" (UID: "c690570f-1cba-450b-bcf6-0549e1ded4b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:38.725749 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:38.725629    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c690570f-1cba-450b-bcf6-0549e1ded4b7-istiod-ca-cert" pod "c690570f-1cba-450b-bcf6-0549e1ded4b7" (UID: "c690570f-1cba-450b-bcf6-0549e1ded4b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:41:38.727647 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:38.727616    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c690570f-1cba-450b-bcf6-0549e1ded4b7-istio-token" pod "c690570f-1cba-450b-bcf6-0549e1ded4b7" (UID: "c690570f-1cba-450b-bcf6-0549e1ded4b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:41:38.727795 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:38.727730    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c690570f-1cba-450b-bcf6-0549e1ded4b7-istio-envoy" pod "c690570f-1cba-450b-bcf6-0549e1ded4b7" (UID: "c690570f-1cba-450b-bcf6-0549e1ded4b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:41:38.727795 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:38.727745    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c690570f-1cba-450b-bcf6-0549e1ded4b7-istio-podinfo" pod "c690570f-1cba-450b-bcf6-0549e1ded4b7" (UID: "c690570f-1cba-450b-bcf6-0549e1ded4b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:41:38.733508 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:38.733478    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c690570f-1cba-450b-bcf6-0549e1ded4b7-kube-api-access-v7nxp" pod "c690570f-1cba-450b-bcf6-0549e1ded4b7" (UID: "c690570f-1cba-450b-bcf6-0549e1ded4b7"). InnerVolumeSpecName "kube-api-access-v7nxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:41:38.826211 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:38.825925    4433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c690570f-1cba-450b-bcf6-0549e1ded4b7-istio-data\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:41:38.826211 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:38.825962    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7nxp\" (UniqueName: \"kubernetes.io/projected/c690570f-1cba-450b-bcf6-0549e1ded4b7-kube-api-access-v7nxp\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:41:38.826211 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:38.825977    4433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c690570f-1cba-450b-bcf6-0549e1ded4b7-workload-certs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:41:38.826211 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:38.825990    4433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c690570f-1cba-450b-bcf6-0549e1ded4b7-istio-envoy\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:41:38.826211 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:38.826003    4433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c690570f-1cba-450b-bcf6-0549e1ded4b7-istio-podinfo\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:41:38.826211 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:38.826014    4433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c690570f-1cba-450b-bcf6-0549e1ded4b7-istiod-ca-cert\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:41:38.826211 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:38.826025    4433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c690570f-1cba-450b-bcf6-0549e1ded4b7-istio-token\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:41:38.826211 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:38.826038    4433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c690570f-1cba-450b-bcf6-0549e1ded4b7-workload-socket\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:41:39.044140 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:39.044093    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"]
Oct 30 17:41:39.056473 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:39.056436    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-575s6"]
Oct 30 17:41:39.697785 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:39.697739    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c690570f-1cba-450b-bcf6-0549e1ded4b7" path="/var/lib/kubelet/pods/c690570f-1cba-450b-bcf6-0549e1ded4b7/volumes"
Oct 30 17:41:56.029578 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:56.029493    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ef70ed1-qlxbp"]
Oct 30 17:41:56.040890 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:56.040850    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ef70ed1-qlxbp"]
Oct 30 17:41:56.976945 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:41:56.976906    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:41:57.694721 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:41:57.694685    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea0c2df3-91dd-40ae-8c3b-5588b8dc3517" path="/var/lib/kubelet/pods/ea0c2df3-91dd-40ae-8c3b-5588b8dc3517/volumes"
Oct 30 17:42:00.028848 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:42:00.028804    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69083163-cvdt5"]
Oct 30 17:42:00.040446 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:42:00.040398    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69083163-cvdt5"]
Oct 30 17:42:01.693555 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:42:01.693511    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1d18482-74d1-4fe0-ae9c-08d49509ae7b" path="/var/lib/kubelet/pods/e1d18482-74d1-4fe0-ae9c-08d49509ae7b/volumes"
Oct 30 17:42:26.977908 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:42:26.977866    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:42:32.029546 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:42:32.029500    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a08570b-9wfk6"]
Oct 30 17:42:32.040534 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:42:32.040504    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a08570b-9wfk6"]
Oct 30 17:42:33.693934 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:42:33.693900    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20dc57cb-856d-4cc9-876e-cfe562599529" path="/var/lib/kubelet/pods/20dc57cb-856d-4cc9-876e-cfe562599529/volumes"
Oct 30 17:42:46.335228 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:42:46.335189    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:42:56.977764 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:42:56.977724    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:43:03.447456 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:03.447421    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:43:06.257593 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:06.257553    4433 scope.go:122] "RemoveContainer" containerID="4247a06bfbc0a6a77b33f186566c80d7a1be6385bcaf48e1c59c588adc5921de"
Oct 30 17:43:06.265864 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:06.265839    4433 scope.go:122] "RemoveContainer" containerID="7867db29edcbd7a4aaefcc211a3affcf434a603d25ca38392f1c933ce18b61ff"
Oct 30 17:43:06.273521 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:06.273498    4433 scope.go:122] "RemoveContainer" containerID="03b7d814830ed0adae13182800f70bf07fe0f2ee2748bc46579d41d59cf09031"
Oct 30 17:43:07.961081 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:07.961032    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:43:09.847295 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847255    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"]
Oct 30 17:43:09.847737 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847648    4433 cpu_manager.go:414] "containerMap: removing container" podUID="9f851311-31f7-477c-a37c-024f63ea7fcf" containerName="istio-proxy"
Oct 30 17:43:09.847737 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847661    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f851311-31f7-477c-a37c-024f63ea7fcf" containerName="istio-proxy"
Oct 30 17:43:09.847737 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847673    4433 cpu_manager.go:414] "containerMap: removing container" podUID="aefefaea-deba-44ec-a357-4fb4ba5e7f6e" containerName="istio-proxy"
Oct 30 17:43:09.847737 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847679    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aefefaea-deba-44ec-a357-4fb4ba5e7f6e" containerName="istio-proxy"
Oct 30 17:43:09.847737 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847689    4433 cpu_manager.go:414] "containerMap: removing container" podUID="8b410f33-6553-4f8d-972f-b21558c4db1a" containerName="istio-proxy"
Oct 30 17:43:09.847737 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847694    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b410f33-6553-4f8d-972f-b21558c4db1a" containerName="istio-proxy"
Oct 30 17:43:09.847737 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847702    4433 cpu_manager.go:414] "containerMap: removing container" podUID="c690570f-1cba-450b-bcf6-0549e1ded4b7" containerName="istio-proxy"
Oct 30 17:43:09.847737 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847708    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c690570f-1cba-450b-bcf6-0549e1ded4b7" containerName="istio-proxy"
Oct 30 17:43:09.847737 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847714    4433 cpu_manager.go:414] "containerMap: removing container" podUID="458eba2e-eb03-452a-bcf4-a5e91da83acf" containerName="main"
Oct 30 17:43:09.847737 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847719    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="458eba2e-eb03-452a-bcf4-a5e91da83acf" containerName="main"
Oct 30 17:43:09.847737 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847726    4433 cpu_manager.go:414] "containerMap: removing container" podUID="8b410f33-6553-4f8d-972f-b21558c4db1a" containerName="istio-proxy"
Oct 30 17:43:09.847737 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847731    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b410f33-6553-4f8d-972f-b21558c4db1a" containerName="istio-proxy"
Oct 30 17:43:09.847737 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847738    4433 cpu_manager.go:414] "containerMap: removing container" podUID="186d5a68-3c34-4953-8e2b-66a1278b9dd7" containerName="maas-api"
Oct 30 17:43:09.847737 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847742    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="186d5a68-3c34-4953-8e2b-66a1278b9dd7" containerName="maas-api"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847753    4433 cpu_manager.go:414] "containerMap: removing container" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847758    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847770    4433 cpu_manager.go:414] "containerMap: removing container" podUID="df7cd3d7-ced4-415b-b6c8-eb31a545daac" containerName="istio-proxy"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847774    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df7cd3d7-ced4-415b-b6c8-eb31a545daac" containerName="istio-proxy"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847790    4433 cpu_manager.go:414] "containerMap: removing container" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847795    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847801    4433 cpu_manager.go:414] "containerMap: removing container" podUID="aefefaea-deba-44ec-a357-4fb4ba5e7f6e" containerName="istio-proxy"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847806    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aefefaea-deba-44ec-a357-4fb4ba5e7f6e" containerName="istio-proxy"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847813    4433 cpu_manager.go:414] "containerMap: removing container" podUID="73875344-e1af-4208-86a8-681a05925c66" containerName="maas-api"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847817    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73875344-e1af-4208-86a8-681a05925c66" containerName="maas-api"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847824    4433 cpu_manager.go:414] "containerMap: removing container" podUID="20dc57cb-856d-4cc9-876e-cfe562599529" containerName="revoke-keys"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847828    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20dc57cb-856d-4cc9-876e-cfe562599529" containerName="revoke-keys"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847834    4433 cpu_manager.go:414] "containerMap: removing container" podUID="51cf9cfa-eed9-4472-ba73-d696ef7bbfcf" containerName="istio-proxy"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847838    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51cf9cfa-eed9-4472-ba73-d696ef7bbfcf" containerName="istio-proxy"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847844    4433 cpu_manager.go:414] "containerMap: removing container" podUID="1a71489d-ed10-4d2e-93df-d5770efaf301" containerName="istio-proxy"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847848    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a71489d-ed10-4d2e-93df-d5770efaf301" containerName="istio-proxy"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847862    4433 cpu_manager.go:414] "containerMap: removing container" podUID="d74c8b0b-844e-493f-a994-5b83b7b73589" containerName="istio-proxy"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847867    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d74c8b0b-844e-493f-a994-5b83b7b73589" containerName="istio-proxy"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847875    4433 cpu_manager.go:414] "containerMap: removing container" podUID="d74c8b0b-844e-493f-a994-5b83b7b73589" containerName="istio-proxy"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847880    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d74c8b0b-844e-493f-a994-5b83b7b73589" containerName="istio-proxy"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847954    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb78a5ca-3157-4707-a75f-f38748185b81" containerName="main"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847962    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="aefefaea-deba-44ec-a357-4fb4ba5e7f6e" containerName="istio-proxy"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847967    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="73875344-e1af-4208-86a8-681a05925c66" containerName="maas-api"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847973    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="20dc57cb-856d-4cc9-876e-cfe562599529" containerName="revoke-keys"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847977    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="51cf9cfa-eed9-4472-ba73-d696ef7bbfcf" containerName="istio-proxy"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847983    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a71489d-ed10-4d2e-93df-d5770efaf301" containerName="istio-proxy"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847989    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="d74c8b0b-844e-493f-a994-5b83b7b73589" containerName="istio-proxy"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.847994    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f851311-31f7-477c-a37c-024f63ea7fcf" containerName="istio-proxy"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.848000    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="aefefaea-deba-44ec-a357-4fb4ba5e7f6e" containerName="istio-proxy"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.848005    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b410f33-6553-4f8d-972f-b21558c4db1a" containerName="istio-proxy"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.848010    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="c690570f-1cba-450b-bcf6-0549e1ded4b7" containerName="istio-proxy"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.848016    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="458eba2e-eb03-452a-bcf4-a5e91da83acf" containerName="main"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.848021    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b410f33-6553-4f8d-972f-b21558c4db1a" containerName="istio-proxy"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.848026    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="186d5a68-3c34-4953-8e2b-66a1278b9dd7" containerName="maas-api"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.848032    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="c99f0236-6725-41a4-abd1-0bc0d9573910" containerName="main"
Oct 30 17:43:09.848198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.848038    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="df7cd3d7-ced4-415b-b6c8-eb31a545daac" containerName="istio-proxy"
Oct 30 17:43:09.851811 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.851782    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"
Oct 30 17:43:09.888713 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.888675    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"]
Oct 30 17:43:09.901116 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.901067    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"]
Oct 30 17:43:09.901823 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.901799    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="d74c8b0b-844e-493f-a994-5b83b7b73589" containerName="istio-proxy"
Oct 30 17:43:09.905939 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.905916    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"
Oct 30 17:43:09.935154 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.935119    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"]
Oct 30 17:43:09.958346 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.958311    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bef99f8d-c7bf-4206-960a-50c84ec6d79d-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dfcvz\" (UID: \"bef99f8d-c7bf-4206-960a-50c84ec6d79d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"
Oct 30 17:43:09.958346 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.958353    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bef99f8d-c7bf-4206-960a-50c84ec6d79d-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dfcvz\" (UID: \"bef99f8d-c7bf-4206-960a-50c84ec6d79d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"
Oct 30 17:43:09.958616 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.958493    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bef99f8d-c7bf-4206-960a-50c84ec6d79d-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dfcvz\" (UID: \"bef99f8d-c7bf-4206-960a-50c84ec6d79d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"
Oct 30 17:43:09.958616 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.958575    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef99f8d-c7bf-4206-960a-50c84ec6d79d-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dfcvz\" (UID: \"bef99f8d-c7bf-4206-960a-50c84ec6d79d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"
Oct 30 17:43:09.958714 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.958645    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c47f8e-91a0-4b8c-8689-8eb97f69cc7a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jpwg6\" (UID: \"30c47f8e-91a0-4b8c-8689-8eb97f69cc7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"
Oct 30 17:43:09.958714 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.958682    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c47f8e-91a0-4b8c-8689-8eb97f69cc7a-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jpwg6\" (UID: \"30c47f8e-91a0-4b8c-8689-8eb97f69cc7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"
Oct 30 17:43:09.958799 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.958784    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxcf6\" (UniqueName: \"kubernetes.io/projected/bef99f8d-c7bf-4206-960a-50c84ec6d79d-kube-api-access-gxcf6\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dfcvz\" (UID: \"bef99f8d-c7bf-4206-960a-50c84ec6d79d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"
Oct 30 17:43:09.958842 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.958817    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bef99f8d-c7bf-4206-960a-50c84ec6d79d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dfcvz\" (UID: \"bef99f8d-c7bf-4206-960a-50c84ec6d79d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"
Oct 30 17:43:09.958885 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.958843    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c47f8e-91a0-4b8c-8689-8eb97f69cc7a-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jpwg6\" (UID: \"30c47f8e-91a0-4b8c-8689-8eb97f69cc7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"
Oct 30 17:43:09.958971 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.958955    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c47f8e-91a0-4b8c-8689-8eb97f69cc7a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jpwg6\" (UID: \"30c47f8e-91a0-4b8c-8689-8eb97f69cc7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"
Oct 30 17:43:09.959020 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.958988    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bef99f8d-c7bf-4206-960a-50c84ec6d79d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dfcvz\" (UID: \"bef99f8d-c7bf-4206-960a-50c84ec6d79d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"
Oct 30 17:43:09.959107 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.959091    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxgb7\" (UniqueName: \"kubernetes.io/projected/30c47f8e-91a0-4b8c-8689-8eb97f69cc7a-kube-api-access-wxgb7\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jpwg6\" (UID: \"30c47f8e-91a0-4b8c-8689-8eb97f69cc7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"
Oct 30 17:43:09.959153 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.959121    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bef99f8d-c7bf-4206-960a-50c84ec6d79d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dfcvz\" (UID: \"bef99f8d-c7bf-4206-960a-50c84ec6d79d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"
Oct 30 17:43:09.959196 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.959153    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c47f8e-91a0-4b8c-8689-8eb97f69cc7a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jpwg6\" (UID: \"30c47f8e-91a0-4b8c-8689-8eb97f69cc7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"
Oct 30 17:43:09.959255 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.959229    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c47f8e-91a0-4b8c-8689-8eb97f69cc7a-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jpwg6\" (UID: \"30c47f8e-91a0-4b8c-8689-8eb97f69cc7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"
Oct 30 17:43:09.959304 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.959278    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c47f8e-91a0-4b8c-8689-8eb97f69cc7a-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jpwg6\" (UID: \"30c47f8e-91a0-4b8c-8689-8eb97f69cc7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"
Oct 30 17:43:09.959380 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.959361    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef99f8d-c7bf-4206-960a-50c84ec6d79d-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dfcvz\" (UID: \"bef99f8d-c7bf-4206-960a-50c84ec6d79d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"
Oct 30 17:43:09.959456 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:09.959435    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c47f8e-91a0-4b8c-8689-8eb97f69cc7a-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jpwg6\" (UID: \"30c47f8e-91a0-4b8c-8689-8eb97f69cc7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"
Oct 30 17:43:10.060926 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.060885    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c47f8e-91a0-4b8c-8689-8eb97f69cc7a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jpwg6\" (UID: \"30c47f8e-91a0-4b8c-8689-8eb97f69cc7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"
Oct 30 17:43:10.060926 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.060930    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c47f8e-91a0-4b8c-8689-8eb97f69cc7a-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jpwg6\" (UID: \"30c47f8e-91a0-4b8c-8689-8eb97f69cc7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"
Oct 30 17:43:10.061219 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.060962    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxcf6\" (UniqueName: \"kubernetes.io/projected/bef99f8d-c7bf-4206-960a-50c84ec6d79d-kube-api-access-gxcf6\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dfcvz\" (UID: \"bef99f8d-c7bf-4206-960a-50c84ec6d79d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"
Oct 30 17:43:10.061219 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.060989    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bef99f8d-c7bf-4206-960a-50c84ec6d79d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dfcvz\" (UID: \"bef99f8d-c7bf-4206-960a-50c84ec6d79d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"
Oct 30 17:43:10.061219 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.061011    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c47f8e-91a0-4b8c-8689-8eb97f69cc7a-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jpwg6\" (UID: \"30c47f8e-91a0-4b8c-8689-8eb97f69cc7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"
Oct 30 17:43:10.061219 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.061043    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c47f8e-91a0-4b8c-8689-8eb97f69cc7a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jpwg6\" (UID: \"30c47f8e-91a0-4b8c-8689-8eb97f69cc7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"
Oct 30 17:43:10.061219 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.061069    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bef99f8d-c7bf-4206-960a-50c84ec6d79d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dfcvz\" (UID: \"bef99f8d-c7bf-4206-960a-50c84ec6d79d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"
Oct 30 17:43:10.061219 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.061116    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxgb7\" (UniqueName: \"kubernetes.io/projected/30c47f8e-91a0-4b8c-8689-8eb97f69cc7a-kube-api-access-wxgb7\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jpwg6\" (UID: \"30c47f8e-91a0-4b8c-8689-8eb97f69cc7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"
Oct 30 17:43:10.061219 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.061139    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bef99f8d-c7bf-4206-960a-50c84ec6d79d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dfcvz\" (UID: \"bef99f8d-c7bf-4206-960a-50c84ec6d79d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"
Oct 30 17:43:10.061219 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.061166    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c47f8e-91a0-4b8c-8689-8eb97f69cc7a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jpwg6\" (UID: \"30c47f8e-91a0-4b8c-8689-8eb97f69cc7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"
Oct 30 17:43:10.061219 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.061194    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c47f8e-91a0-4b8c-8689-8eb97f69cc7a-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jpwg6\" (UID: \"30c47f8e-91a0-4b8c-8689-8eb97f69cc7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"
Oct 30 17:43:10.061219 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.061216    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c47f8e-91a0-4b8c-8689-8eb97f69cc7a-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jpwg6\" (UID: \"30c47f8e-91a0-4b8c-8689-8eb97f69cc7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"
Oct 30 17:43:10.061815 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.061259    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef99f8d-c7bf-4206-960a-50c84ec6d79d-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dfcvz\" (UID: \"bef99f8d-c7bf-4206-960a-50c84ec6d79d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"
Oct 30 17:43:10.061815 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.061301    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c47f8e-91a0-4b8c-8689-8eb97f69cc7a-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jpwg6\" (UID: \"30c47f8e-91a0-4b8c-8689-8eb97f69cc7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"
Oct 30 17:43:10.061815 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.061348    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bef99f8d-c7bf-4206-960a-50c84ec6d79d-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dfcvz\" (UID: \"bef99f8d-c7bf-4206-960a-50c84ec6d79d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"
Oct 30 17:43:10.061815 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.061372    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bef99f8d-c7bf-4206-960a-50c84ec6d79d-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dfcvz\" (UID: \"bef99f8d-c7bf-4206-960a-50c84ec6d79d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"
Oct 30 17:43:10.061815 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.061430    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bef99f8d-c7bf-4206-960a-50c84ec6d79d-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dfcvz\" (UID: \"bef99f8d-c7bf-4206-960a-50c84ec6d79d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"
Oct 30 17:43:10.061815 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.061461    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef99f8d-c7bf-4206-960a-50c84ec6d79d-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dfcvz\" (UID: \"bef99f8d-c7bf-4206-960a-50c84ec6d79d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"
Oct 30 17:43:10.061815 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.061575    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c47f8e-91a0-4b8c-8689-8eb97f69cc7a-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jpwg6\" (UID: \"30c47f8e-91a0-4b8c-8689-8eb97f69cc7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"
Oct 30 17:43:10.062248 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.061834    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef99f8d-c7bf-4206-960a-50c84ec6d79d-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dfcvz\" (UID: \"bef99f8d-c7bf-4206-960a-50c84ec6d79d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"
Oct 30 17:43:10.062248 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.061865    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c47f8e-91a0-4b8c-8689-8eb97f69cc7a-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jpwg6\" (UID: \"30c47f8e-91a0-4b8c-8689-8eb97f69cc7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"
Oct 30 17:43:10.062248 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.062099    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c47f8e-91a0-4b8c-8689-8eb97f69cc7a-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jpwg6\" (UID: \"30c47f8e-91a0-4b8c-8689-8eb97f69cc7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"
Oct 30 17:43:10.062449 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.062340    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef99f8d-c7bf-4206-960a-50c84ec6d79d-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dfcvz\" (UID: \"bef99f8d-c7bf-4206-960a-50c84ec6d79d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"
Oct 30 17:43:10.062810 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.062779    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bef99f8d-c7bf-4206-960a-50c84ec6d79d-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dfcvz\" (UID: \"bef99f8d-c7bf-4206-960a-50c84ec6d79d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"
Oct 30 17:43:10.062925 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.062854    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bef99f8d-c7bf-4206-960a-50c84ec6d79d-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dfcvz\" (UID: \"bef99f8d-c7bf-4206-960a-50c84ec6d79d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"
Oct 30 17:43:10.063632 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.063555    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bef99f8d-c7bf-4206-960a-50c84ec6d79d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dfcvz\" (UID: \"bef99f8d-c7bf-4206-960a-50c84ec6d79d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"
Oct 30 17:43:10.064323 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.064279    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bef99f8d-c7bf-4206-960a-50c84ec6d79d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dfcvz\" (UID: \"bef99f8d-c7bf-4206-960a-50c84ec6d79d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"
Oct 30 17:43:10.064658 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.064630    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c47f8e-91a0-4b8c-8689-8eb97f69cc7a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jpwg6\" (UID: \"30c47f8e-91a0-4b8c-8689-8eb97f69cc7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"
Oct 30 17:43:10.064849 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.064825    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c47f8e-91a0-4b8c-8689-8eb97f69cc7a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jpwg6\" (UID: \"30c47f8e-91a0-4b8c-8689-8eb97f69cc7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"
Oct 30 17:43:10.064932 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.064864    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c47f8e-91a0-4b8c-8689-8eb97f69cc7a-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jpwg6\" (UID: \"30c47f8e-91a0-4b8c-8689-8eb97f69cc7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"
Oct 30 17:43:10.065631 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.065603    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bef99f8d-c7bf-4206-960a-50c84ec6d79d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dfcvz\" (UID: \"bef99f8d-c7bf-4206-960a-50c84ec6d79d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"
Oct 30 17:43:10.066243 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.066218    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c47f8e-91a0-4b8c-8689-8eb97f69cc7a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jpwg6\" (UID: \"30c47f8e-91a0-4b8c-8689-8eb97f69cc7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"
Oct 30 17:43:10.084917 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.084878    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c47f8e-91a0-4b8c-8689-8eb97f69cc7a-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jpwg6\" (UID: \"30c47f8e-91a0-4b8c-8689-8eb97f69cc7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"
Oct 30 17:43:10.085256 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.085222    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bef99f8d-c7bf-4206-960a-50c84ec6d79d-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dfcvz\" (UID: \"bef99f8d-c7bf-4206-960a-50c84ec6d79d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"
Oct 30 17:43:10.085378 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.085339    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxgb7\" (UniqueName: \"kubernetes.io/projected/30c47f8e-91a0-4b8c-8689-8eb97f69cc7a-kube-api-access-wxgb7\") pod \"maas-default-gateway-openshift-default-6db586dd6c-jpwg6\" (UID: \"30c47f8e-91a0-4b8c-8689-8eb97f69cc7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"
Oct 30 17:43:10.087570 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.087542    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxcf6\" (UniqueName: \"kubernetes.io/projected/bef99f8d-c7bf-4206-960a-50c84ec6d79d-kube-api-access-gxcf6\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dfcvz\" (UID: \"bef99f8d-c7bf-4206-960a-50c84ec6d79d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"
Oct 30 17:43:10.168861 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.168757    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"
Oct 30 17:43:10.227248 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.227212    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"
Oct 30 17:43:10.496988 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.496949    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"]
Oct 30 17:43:10.500215 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:43:10.500188    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef99f8d_c7bf_4206_960a_50c84ec6d79d.slice/crio-7042ce227c18e5602ee7e6015e5e50c69e3ab259da4a5b5a3c5a1ac9b05da102 WatchSource:0}: Error finding container 7042ce227c18e5602ee7e6015e5e50c69e3ab259da4a5b5a3c5a1ac9b05da102: Status 404 returned error can't find the container with id 7042ce227c18e5602ee7e6015e5e50c69e3ab259da4a5b5a3c5a1ac9b05da102
Oct 30 17:43:10.502331 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.502302    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:43:10.502386 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.502368    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:43:10.502438 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.502398    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:43:10.557309 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.557280    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"]
Oct 30 17:43:10.560309 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:43:10.560276    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c47f8e_91a0_4b8c_8689_8eb97f69cc7a.slice/crio-d43b2e55a9264e25fb08d9afa8c54e2ec410b4e0068dd2a24641b97ef35054d0 WatchSource:0}: Error finding container d43b2e55a9264e25fb08d9afa8c54e2ec410b4e0068dd2a24641b97ef35054d0: Status 404 returned error can't find the container with id d43b2e55a9264e25fb08d9afa8c54e2ec410b4e0068dd2a24641b97ef35054d0
Oct 30 17:43:10.562497 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.562469    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:43:10.562579 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.562529    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:43:10.562579 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:10.562561    4433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 17:43:11.134527 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:11.134484    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6" event={"ID":"30c47f8e-91a0-4b8c-8689-8eb97f69cc7a","Type":"ContainerStarted","Data":"3a6a80d9bcc65a27c150d1fb967b06734c2613bdc76efa1ca61e7300ac7f6a9f"}
Oct 30 17:43:11.134968 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:11.134576    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6" event={"ID":"30c47f8e-91a0-4b8c-8689-8eb97f69cc7a","Type":"ContainerStarted","Data":"d43b2e55a9264e25fb08d9afa8c54e2ec410b4e0068dd2a24641b97ef35054d0"}
Oct 30 17:43:11.136110 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:11.136086    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz" event={"ID":"bef99f8d-c7bf-4206-960a-50c84ec6d79d","Type":"ContainerStarted","Data":"a2b90f3a0dc0526bbe0c27817d6604621b0c1685e75f8c20eef5fc156405c60c"}
Oct 30 17:43:11.136229 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:11.136126    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz" event={"ID":"bef99f8d-c7bf-4206-960a-50c84ec6d79d","Type":"ContainerStarted","Data":"7042ce227c18e5602ee7e6015e5e50c69e3ab259da4a5b5a3c5a1ac9b05da102"}
Oct 30 17:43:11.173440 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:11.172558    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"
Oct 30 17:43:11.173440 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:11.172696    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dfcvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 17:43:11.173440 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:11.172736    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz" podUID="bef99f8d-c7bf-4206-960a-50c84ec6d79d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 17:43:11.227977 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:11.227935    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"
Oct 30 17:43:11.229346 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:11.229322    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-jpwg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 17:43:11.229486 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:11.229364    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6" podUID="30c47f8e-91a0-4b8c-8689-8eb97f69cc7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 17:43:11.239108 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:11.239065    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6" podStartSLOduration=2.239052408 podStartE2EDuration="2.239052408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:43:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:43:11.192594008 +0000 UTC m=+3367.627496239" watchObservedRunningTime="2026-10-30 17:43:11.239052408 +0000 UTC m=+3367.673954696"
Oct 30 17:43:11.240144 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:11.240107    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz" podStartSLOduration=2.240095844 podStartE2EDuration="2.240095844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:43:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:43:11.238362527 +0000 UTC m=+3367.673264734" watchObservedRunningTime="2026-10-30 17:43:11.240095844 +0000 UTC m=+3367.674998051"
Oct 30 17:43:12.169915 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:12.169872    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dfcvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 17:43:12.171426 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:12.171020    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz" podUID="bef99f8d-c7bf-4206-960a-50c84ec6d79d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 17:43:12.227839 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:12.227804    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-jpwg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 17:43:12.228021 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:12.227853    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6" podUID="30c47f8e-91a0-4b8c-8689-8eb97f69cc7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 17:43:13.173428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:13.172862    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dfcvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 17:43:13.173428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:13.172928    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz" podUID="bef99f8d-c7bf-4206-960a-50c84ec6d79d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 17:43:13.229499 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:13.228471    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-jpwg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 17:43:13.229499 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:13.228535    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6" podUID="30c47f8e-91a0-4b8c-8689-8eb97f69cc7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 17:43:14.170946 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:14.170900    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dfcvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 17:43:14.171222 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:14.171192    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz" podUID="bef99f8d-c7bf-4206-960a-50c84ec6d79d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 17:43:14.228042 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:14.228010    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-jpwg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 17:43:14.228489 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:14.228394    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6" podUID="30c47f8e-91a0-4b8c-8689-8eb97f69cc7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 17:43:15.188568 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:15.188526    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dfcvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 17:43:15.188898 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:15.188863    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz" podUID="bef99f8d-c7bf-4206-960a-50c84ec6d79d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 17:43:15.227733 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:15.227700    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-jpwg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 17:43:15.227919 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:15.227751    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6" podUID="30c47f8e-91a0-4b8c-8689-8eb97f69cc7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 17:43:16.170691 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:16.170172    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dfcvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 17:43:16.170691 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:16.170237    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz" podUID="bef99f8d-c7bf-4206-960a-50c84ec6d79d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 17:43:16.227769 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:16.227731    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-jpwg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 17:43:16.227933 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:16.227782    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6" podUID="30c47f8e-91a0-4b8c-8689-8eb97f69cc7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 17:43:17.170011 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:17.169968    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dfcvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 17:43:17.170287 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:17.170038    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz" podUID="bef99f8d-c7bf-4206-960a-50c84ec6d79d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 17:43:17.228151 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:17.228114    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-jpwg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 17:43:17.228569 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:17.228168    4433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6" podUID="30c47f8e-91a0-4b8c-8689-8eb97f69cc7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 17:43:18.827171 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:18.827130    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"
Oct 30 17:43:18.827910 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:18.827888    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"
Oct 30 17:43:18.828837 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:18.828809    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz"
Oct 30 17:43:18.902588 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:18.902548    4433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"
Oct 30 17:43:18.902915 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:18.902892    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"
Oct 30 17:43:18.903822 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:18.903799    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6"
Oct 30 17:43:23.560798 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:23.560755    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:43:26.982424 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:43:26.982370    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:43:39.566452 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:43:39.566394    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:43:56.977424 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:43:56.977370    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:44:03.057244 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:44:03.057206    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:44:06.303415 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:44:06.303380    4433 scope.go:122] "RemoveContainer" containerID="5d4e7ca2fb7b549882ace77606f112f44107e2a4669635db496f5ed0705adea3"
Oct 30 17:44:13.348433 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:44:13.348384    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:44:17.715655 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:44:17.715614    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:44:19.946593 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:44:19.946552    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-jpwg6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:44:19.946991 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:44:19.946627    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6" podUID="30c47f8e-91a0-4b8c-8689-8eb97f69cc7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:44:26.976799 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:44:26.976763    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:44:35.034643 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:44:35.034604    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:44:40.297429 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:44:40.296455    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:44:56.625984 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:44:56.625890    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:44:56.977821 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:44:56.977776    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:45:00.959447 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:00.959399    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:45:05.544655 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:05.544611    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-llkcw"]
Oct 30 17:45:05.548539 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:05.548516    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-llkcw"
Oct 30 17:45:05.563005 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:05.562971    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:45:05.574176 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:05.574149    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 17:45:05.574176 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:05.574149    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\""
Oct 30 17:45:05.574341 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:05.574194    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 17:45:05.576063 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:05.576038    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-llkcw"]
Oct 30 17:45:05.694221 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:05.694177    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrq9b\" (UniqueName: \"kubernetes.io/projected/8ed7f4ff-c815-41a7-b130-c75651696aa7-kube-api-access-rrq9b\") pod \"maas-api-revoke-sub-79805ca9-llkcw\" (UID: \"8ed7f4ff-c815-41a7-b130-c75651696aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-llkcw"
Oct 30 17:45:05.694433 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:05.694296    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ed7f4ff-c815-41a7-b130-c75651696aa7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-llkcw\" (UID: \"8ed7f4ff-c815-41a7-b130-c75651696aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-llkcw"
Oct 30 17:45:05.694433 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:05.694349    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ed7f4ff-c815-41a7-b130-c75651696aa7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-llkcw\" (UID: \"8ed7f4ff-c815-41a7-b130-c75651696aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-llkcw"
Oct 30 17:45:05.795710 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:05.795619    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ed7f4ff-c815-41a7-b130-c75651696aa7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-llkcw\" (UID: \"8ed7f4ff-c815-41a7-b130-c75651696aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-llkcw"
Oct 30 17:45:05.795710 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:05.795680    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ed7f4ff-c815-41a7-b130-c75651696aa7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-llkcw\" (UID: \"8ed7f4ff-c815-41a7-b130-c75651696aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-llkcw"
Oct 30 17:45:05.795927 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:05.795752    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrq9b\" (UniqueName: \"kubernetes.io/projected/8ed7f4ff-c815-41a7-b130-c75651696aa7-kube-api-access-rrq9b\") pod \"maas-api-revoke-sub-79805ca9-llkcw\" (UID: \"8ed7f4ff-c815-41a7-b130-c75651696aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-llkcw"
Oct 30 17:45:05.796428 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:05.796383    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ed7f4ff-c815-41a7-b130-c75651696aa7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-llkcw\" (UID: \"8ed7f4ff-c815-41a7-b130-c75651696aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-llkcw"
Oct 30 17:45:05.798044 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:05.798024    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ed7f4ff-c815-41a7-b130-c75651696aa7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-llkcw\" (UID: \"8ed7f4ff-c815-41a7-b130-c75651696aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-llkcw"
Oct 30 17:45:05.832597 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:05.832567    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrq9b\" (UniqueName: \"kubernetes.io/projected/8ed7f4ff-c815-41a7-b130-c75651696aa7-kube-api-access-rrq9b\") pod \"maas-api-revoke-sub-79805ca9-llkcw\" (UID: \"8ed7f4ff-c815-41a7-b130-c75651696aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-llkcw"
Oct 30 17:45:05.861653 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:05.861623    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-llkcw"
Oct 30 17:45:06.159076 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:06.158999    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-llkcw"]
Oct 30 17:45:06.163560 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:45:06.163532    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ed7f4ff_c815_41a7_b130_c75651696aa7.slice/crio-67e87095c5396257e132b3f49dc2ea49fe842d8f501aac724d867b902c9fb709 WatchSource:0}: Error finding container 67e87095c5396257e132b3f49dc2ea49fe842d8f501aac724d867b902c9fb709: Status 404 returned error can't find the container with id 67e87095c5396257e132b3f49dc2ea49fe842d8f501aac724d867b902c9fb709
Oct 30 17:45:06.848233 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:06.848134    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-llkcw" event={"ID":"8ed7f4ff-c815-41a7-b130-c75651696aa7","Type":"ContainerStarted","Data":"a6309a3b05aaa44247ce52453526c4416be6b8e9aa6b674fa14fd3ded8ce9acc"}
Oct 30 17:45:06.848233 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:06.848191    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-llkcw" event={"ID":"8ed7f4ff-c815-41a7-b130-c75651696aa7","Type":"ContainerStarted","Data":"67e87095c5396257e132b3f49dc2ea49fe842d8f501aac724d867b902c9fb709"}
Oct 30 17:45:06.889941 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:06.889769    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-llkcw" podStartSLOduration=1.889748647 podStartE2EDuration="1.889748647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:06.888132744 +0000 UTC m=+3483.323034956" watchObservedRunningTime="2026-10-30 17:45:06.889748647 +0000 UTC m=+3483.324650854"
Oct 30 17:45:07.854630 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:07.854590    4433 generic.go:356] "Generic (PLEG): container finished" podID="8ed7f4ff-c815-41a7-b130-c75651696aa7" containerID="a6309a3b05aaa44247ce52453526c4416be6b8e9aa6b674fa14fd3ded8ce9acc" exitCode=0
Oct 30 17:45:07.855047 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:07.854674    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-llkcw" event={"ID":"8ed7f4ff-c815-41a7-b130-c75651696aa7","Type":"ContainerDied","Data":"a6309a3b05aaa44247ce52453526c4416be6b8e9aa6b674fa14fd3ded8ce9acc"}
Oct 30 17:45:08.994157 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:08.994131    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-llkcw"
Oct 30 17:45:09.130251 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:09.130160    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ed7f4ff-c815-41a7-b130-c75651696aa7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ed7f4ff-c815-41a7-b130-c75651696aa7-maas-api-openshift-ca\") pod \"8ed7f4ff-c815-41a7-b130-c75651696aa7\" (UID: \"8ed7f4ff-c815-41a7-b130-c75651696aa7\") "
Oct 30 17:45:09.130251 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:09.130229    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ed7f4ff-c815-41a7-b130-c75651696aa7-kube-api-access-rrq9b\" (UniqueName: \"kubernetes.io/projected/8ed7f4ff-c815-41a7-b130-c75651696aa7-kube-api-access-rrq9b\") pod \"8ed7f4ff-c815-41a7-b130-c75651696aa7\" (UID: \"8ed7f4ff-c815-41a7-b130-c75651696aa7\") "
Oct 30 17:45:09.130494 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:09.130299    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ed7f4ff-c815-41a7-b130-c75651696aa7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ed7f4ff-c815-41a7-b130-c75651696aa7-maas-api-xks-ca\") pod \"8ed7f4ff-c815-41a7-b130-c75651696aa7\" (UID: \"8ed7f4ff-c815-41a7-b130-c75651696aa7\") "
Oct 30 17:45:09.130642 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:09.130613    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ed7f4ff-c815-41a7-b130-c75651696aa7-maas-api-openshift-ca" pod "8ed7f4ff-c815-41a7-b130-c75651696aa7" (UID: "8ed7f4ff-c815-41a7-b130-c75651696aa7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:45:09.132460 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:09.132426    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ed7f4ff-c815-41a7-b130-c75651696aa7-maas-api-xks-ca" pod "8ed7f4ff-c815-41a7-b130-c75651696aa7" (UID: "8ed7f4ff-c815-41a7-b130-c75651696aa7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:45:09.132580 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:09.132547    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ed7f4ff-c815-41a7-b130-c75651696aa7-kube-api-access-rrq9b" pod "8ed7f4ff-c815-41a7-b130-c75651696aa7" (UID: "8ed7f4ff-c815-41a7-b130-c75651696aa7"). InnerVolumeSpecName "kube-api-access-rrq9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:45:09.231437 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:09.231368    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ed7f4ff-c815-41a7-b130-c75651696aa7-maas-api-xks-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:45:09.231437 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:09.231429    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ed7f4ff-c815-41a7-b130-c75651696aa7-maas-api-openshift-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:45:09.231437 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:09.231444    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrq9b\" (UniqueName: \"kubernetes.io/projected/8ed7f4ff-c815-41a7-b130-c75651696aa7-kube-api-access-rrq9b\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:45:09.866171 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:09.866137    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-llkcw"
Oct 30 17:45:09.866342 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:09.866176    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-llkcw" event={"ID":"8ed7f4ff-c815-41a7-b130-c75651696aa7","Type":"ContainerDied","Data":"67e87095c5396257e132b3f49dc2ea49fe842d8f501aac724d867b902c9fb709"}
Oct 30 17:45:09.866342 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:09.866225    4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67e87095c5396257e132b3f49dc2ea49fe842d8f501aac724d867b902c9fb709"
Oct 30 17:45:17.638391 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:17.638353    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:45:26.978263 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:45:26.978224    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:45:33.671437 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:33.671266    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:45:44.529373 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:44.529334    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:45:50.070071 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:45:50.070027    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:45:56.978660 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:45:56.978620    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:46:06.731107 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:46:06.731070    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:46:11.839462 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:46:11.839425    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:46:26.978211 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:46:26.978169    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:46:30.732669 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:46:30.732634    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:46:36.023945 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:46:36.023907    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:46:50.260258 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:46:50.260219    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:46:56.977497 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:46:56.977457    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:47:05.986722 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:47:05.986682    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:47:19.843182 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:47:19.843136    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:47:26.977324 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:47:26.977287    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:47:37.330892 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:47:37.330851    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:47:51.656566 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:47:51.656312    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:47:56.977288 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:47:56.977250    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:48:10.729220 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:48:10.729185    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:48:17.063540 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:48:17.063498    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:48:26.979909 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:48:26.979868    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:48:56.977516 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:48:56.977387    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:49:26.977259 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:49:26.977216    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:49:49.441534 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:49:49.441497    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:49:56.977795 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:49:56.977754    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:50:04.724732 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:50:04.724684    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:50:09.031123 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:50:09.031079    4433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-llkcw"]
Oct 30 17:50:09.042363 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:50:09.042329    4433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-llkcw"]
Oct 30 17:50:09.710064 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:50:09.710023    4433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ed7f4ff-c815-41a7-b130-c75651696aa7" path="/var/lib/kubelet/pods/8ed7f4ff-c815-41a7-b130-c75651696aa7/volumes"
Oct 30 17:50:26.977696 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:50:26.977649    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:50:31.735234 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:50:31.735193    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:50:46.350563 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:50:46.350480    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:50:56.987780 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:50:56.987706    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:51:01.135684 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:51:01.135646    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:51:06.366784 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:51:06.366747    4433 scope.go:122] "RemoveContainer" containerID="a6309a3b05aaa44247ce52453526c4416be6b8e9aa6b674fa14fd3ded8ce9acc"
Oct 30 17:51:14.558647 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:51:14.558604    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:51:26.976747 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:51:26.976702    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:51:27.117781 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:51:27.117744    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:51:40.873856 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:51:40.873820    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:51:57.007620 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:51:57.007576    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:51:58.960456 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:51:58.960417    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:52:13.663919 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:52:13.663872    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:52:16.582635 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:52:16.582540    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n9gd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:52:16.583367 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:52:16.583216    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n9gd9" podUID="c986e3ae-ffab-420f-92ca-8234d2d918f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:52:26.976620 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:52:26.976575    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:52:27.963618 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:52:27.963587    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:52:34.869652 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:52:34.869603    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dfcvz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:52:34.870472 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:52:34.870433    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dfcvz" podUID="bef99f8d-c7bf-4206-960a-50c84ec6d79d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:52:34.948609 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:52:34.948570    4433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-jpwg6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:52:34.948837 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:52:34.948643    4433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-jpwg6" podUID="30c47f8e-91a0-4b8c-8689-8eb97f69cc7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:52:36.349327 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:52:36.349290    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:52:46.932510 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:52:46.932473    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:52:51.261681 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:52:51.261643    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:52:56.977244 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:52:56.977207    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:53:09.266596 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:09.266559    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:53:19.317553 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:19.317516    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:53:26.977231 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:53:26.977185    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:53:27.534297 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:27.534263    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:53:34.372933 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:34.372890    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-njf4t"]
Oct 30 17:53:34.373450 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:34.373296    4433 cpu_manager.go:414] "containerMap: removing container" podUID="8ed7f4ff-c815-41a7-b130-c75651696aa7" containerName="revoke-keys"
Oct 30 17:53:34.373450 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:34.373308    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ed7f4ff-c815-41a7-b130-c75651696aa7" containerName="revoke-keys"
Oct 30 17:53:34.373450 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:34.373366    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ed7f4ff-c815-41a7-b130-c75651696aa7" containerName="revoke-keys"
Oct 30 17:53:34.376508 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:34.376492    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-njf4t"
Oct 30 17:53:34.390583 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:34.390552    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:53:34.401987 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:34.401954    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\""
Oct 30 17:53:34.401987 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:34.401995    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 17:53:34.402209 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:34.402003    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 17:53:34.404895 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:34.404863    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-njf4t"]
Oct 30 17:53:34.511623 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:34.511592    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82nrs\" (UniqueName: \"kubernetes.io/projected/80afb90a-700b-4b28-ab7b-e649e5c9bd01-kube-api-access-82nrs\") pod \"maas-api-revoke-sub-f03629db-njf4t\" (UID: \"80afb90a-700b-4b28-ab7b-e649e5c9bd01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-njf4t"
Oct 30 17:53:34.511829 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:34.511674    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80afb90a-700b-4b28-ab7b-e649e5c9bd01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-njf4t\" (UID: \"80afb90a-700b-4b28-ab7b-e649e5c9bd01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-njf4t"
Oct 30 17:53:34.511829 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:34.511703    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80afb90a-700b-4b28-ab7b-e649e5c9bd01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-njf4t\" (UID: \"80afb90a-700b-4b28-ab7b-e649e5c9bd01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-njf4t"
Oct 30 17:53:34.612991 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:34.612948    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80afb90a-700b-4b28-ab7b-e649e5c9bd01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-njf4t\" (UID: \"80afb90a-700b-4b28-ab7b-e649e5c9bd01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-njf4t"
Oct 30 17:53:34.613198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:34.613008    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80afb90a-700b-4b28-ab7b-e649e5c9bd01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-njf4t\" (UID: \"80afb90a-700b-4b28-ab7b-e649e5c9bd01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-njf4t"
Oct 30 17:53:34.613198 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:34.613101    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82nrs\" (UniqueName: \"kubernetes.io/projected/80afb90a-700b-4b28-ab7b-e649e5c9bd01-kube-api-access-82nrs\") pod \"maas-api-revoke-sub-f03629db-njf4t\" (UID: \"80afb90a-700b-4b28-ab7b-e649e5c9bd01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-njf4t"
Oct 30 17:53:34.613734 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:34.613694    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80afb90a-700b-4b28-ab7b-e649e5c9bd01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-njf4t\" (UID: \"80afb90a-700b-4b28-ab7b-e649e5c9bd01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-njf4t"
Oct 30 17:53:34.615317 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:34.615294    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80afb90a-700b-4b28-ab7b-e649e5c9bd01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-njf4t\" (UID: \"80afb90a-700b-4b28-ab7b-e649e5c9bd01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-njf4t"
Oct 30 17:53:34.636878 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:34.636819    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82nrs\" (UniqueName: \"kubernetes.io/projected/80afb90a-700b-4b28-ab7b-e649e5c9bd01-kube-api-access-82nrs\") pod \"maas-api-revoke-sub-f03629db-njf4t\" (UID: \"80afb90a-700b-4b28-ab7b-e649e5c9bd01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-njf4t"
Oct 30 17:53:34.688976 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:34.688938    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-njf4t"
Oct 30 17:53:34.984309 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:34.984276    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-njf4t"]
Oct 30 17:53:34.987720 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:53:34.987690    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80afb90a_700b_4b28_ab7b_e649e5c9bd01.slice/crio-34f7ab889f2e1d1aa6d070da92644831361baa605a3e5d605cba094ec88a6e87 WatchSource:0}: Error finding container 34f7ab889f2e1d1aa6d070da92644831361baa605a3e5d605cba094ec88a6e87: Status 404 returned error can't find the container with id 34f7ab889f2e1d1aa6d070da92644831361baa605a3e5d605cba094ec88a6e87
Oct 30 17:53:35.520112 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:35.520020    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-njf4t" event={"ID":"80afb90a-700b-4b28-ab7b-e649e5c9bd01","Type":"ContainerStarted","Data":"552c02606aa39aa861f5778c0e339b304f62a6f82d8b6ca4e338eb0f972a3fef"}
Oct 30 17:53:35.520112 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:35.520078    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-njf4t" event={"ID":"80afb90a-700b-4b28-ab7b-e649e5c9bd01","Type":"ContainerStarted","Data":"34f7ab889f2e1d1aa6d070da92644831361baa605a3e5d605cba094ec88a6e87"}
Oct 30 17:53:35.569438 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:35.569373    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-njf4t" podStartSLOduration=1.569357911 podStartE2EDuration="1.569357911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:53:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:53:35.568218783 +0000 UTC m=+3992.003121000" watchObservedRunningTime="2026-10-30 17:53:35.569357911 +0000 UTC m=+3992.004260154"
Oct 30 17:53:36.526121 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:36.526086    4433 generic.go:356] "Generic (PLEG): container finished" podID="80afb90a-700b-4b28-ab7b-e649e5c9bd01" containerID="552c02606aa39aa861f5778c0e339b304f62a6f82d8b6ca4e338eb0f972a3fef" exitCode=0
Oct 30 17:53:36.526574 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:36.526193    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-njf4t" event={"ID":"80afb90a-700b-4b28-ab7b-e649e5c9bd01","Type":"ContainerDied","Data":"552c02606aa39aa861f5778c0e339b304f62a6f82d8b6ca4e338eb0f972a3fef"}
Oct 30 17:53:37.673858 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:37.673830    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-njf4t"
Oct 30 17:53:37.740012 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:37.739977    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80afb90a-700b-4b28-ab7b-e649e5c9bd01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80afb90a-700b-4b28-ab7b-e649e5c9bd01-maas-api-openshift-ca\") pod \"80afb90a-700b-4b28-ab7b-e649e5c9bd01\" (UID: \"80afb90a-700b-4b28-ab7b-e649e5c9bd01\") "
Oct 30 17:53:37.740187 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:37.740102    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80afb90a-700b-4b28-ab7b-e649e5c9bd01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80afb90a-700b-4b28-ab7b-e649e5c9bd01-maas-api-xks-ca\") pod \"80afb90a-700b-4b28-ab7b-e649e5c9bd01\" (UID: \"80afb90a-700b-4b28-ab7b-e649e5c9bd01\") "
Oct 30 17:53:37.740239 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:37.740197    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80afb90a-700b-4b28-ab7b-e649e5c9bd01-kube-api-access-82nrs\" (UniqueName: \"kubernetes.io/projected/80afb90a-700b-4b28-ab7b-e649e5c9bd01-kube-api-access-82nrs\") pod \"80afb90a-700b-4b28-ab7b-e649e5c9bd01\" (UID: \"80afb90a-700b-4b28-ab7b-e649e5c9bd01\") "
Oct 30 17:53:37.740418 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:37.740372    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80afb90a-700b-4b28-ab7b-e649e5c9bd01-maas-api-openshift-ca" pod "80afb90a-700b-4b28-ab7b-e649e5c9bd01" (UID: "80afb90a-700b-4b28-ab7b-e649e5c9bd01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:53:37.740832 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:37.740546    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80afb90a-700b-4b28-ab7b-e649e5c9bd01-maas-api-openshift-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:53:37.742794 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:37.742764    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80afb90a-700b-4b28-ab7b-e649e5c9bd01-maas-api-xks-ca" pod "80afb90a-700b-4b28-ab7b-e649e5c9bd01" (UID: "80afb90a-700b-4b28-ab7b-e649e5c9bd01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:53:37.742970 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:37.742946    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80afb90a-700b-4b28-ab7b-e649e5c9bd01-kube-api-access-82nrs" pod "80afb90a-700b-4b28-ab7b-e649e5c9bd01" (UID: "80afb90a-700b-4b28-ab7b-e649e5c9bd01"). InnerVolumeSpecName "kube-api-access-82nrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:53:37.841916 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:37.841819    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82nrs\" (UniqueName: \"kubernetes.io/projected/80afb90a-700b-4b28-ab7b-e649e5c9bd01-kube-api-access-82nrs\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:53:37.841916 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:37.841858    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80afb90a-700b-4b28-ab7b-e649e5c9bd01-maas-api-xks-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:53:38.536042 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:38.536011    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-njf4t"
Oct 30 17:53:38.536251 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:38.536006    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-njf4t" event={"ID":"80afb90a-700b-4b28-ab7b-e649e5c9bd01","Type":"ContainerDied","Data":"34f7ab889f2e1d1aa6d070da92644831361baa605a3e5d605cba094ec88a6e87"}
Oct 30 17:53:38.536251 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:38.536169    4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34f7ab889f2e1d1aa6d070da92644831361baa605a3e5d605cba094ec88a6e87"
Oct 30 17:53:41.031168 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:41.031131    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:53:46.569675 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:46.569590    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-llmhr"]
Oct 30 17:53:46.570076 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:46.570040    4433 cpu_manager.go:414] "containerMap: removing container" podUID="80afb90a-700b-4b28-ab7b-e649e5c9bd01" containerName="revoke-keys"
Oct 30 17:53:46.570121 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:46.570077    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80afb90a-700b-4b28-ab7b-e649e5c9bd01" containerName="revoke-keys"
Oct 30 17:53:46.570168 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:46.570154    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="80afb90a-700b-4b28-ab7b-e649e5c9bd01" containerName="revoke-keys"
Oct 30 17:53:46.573382 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:46.573362    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-llmhr"
Oct 30 17:53:46.588056 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:46.588022    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 17:53:46.599384 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:46.599350    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:53:46.600128 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:46.600099    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 17:53:46.600250 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:46.600204    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\""
Oct 30 17:53:46.602366 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:46.602341    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-llmhr"]
Oct 30 17:53:46.739781 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:46.739744    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp9lg\" (UniqueName: \"kubernetes.io/projected/b2203d97-4798-4cf0-84ce-94076c2568eb-kube-api-access-rp9lg\") pod \"maas-api-revoke-sub-0165d7d2-llmhr\" (UID: \"b2203d97-4798-4cf0-84ce-94076c2568eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-llmhr"
Oct 30 17:53:46.739781 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:46.739784    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2203d97-4798-4cf0-84ce-94076c2568eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-llmhr\" (UID: \"b2203d97-4798-4cf0-84ce-94076c2568eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-llmhr"
Oct 30 17:53:46.740013 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:46.739876    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2203d97-4798-4cf0-84ce-94076c2568eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-llmhr\" (UID: \"b2203d97-4798-4cf0-84ce-94076c2568eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-llmhr"
Oct 30 17:53:46.841058 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:46.840960    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp9lg\" (UniqueName: \"kubernetes.io/projected/b2203d97-4798-4cf0-84ce-94076c2568eb-kube-api-access-rp9lg\") pod \"maas-api-revoke-sub-0165d7d2-llmhr\" (UID: \"b2203d97-4798-4cf0-84ce-94076c2568eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-llmhr"
Oct 30 17:53:46.841058 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:46.841004    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2203d97-4798-4cf0-84ce-94076c2568eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-llmhr\" (UID: \"b2203d97-4798-4cf0-84ce-94076c2568eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-llmhr"
Oct 30 17:53:46.841283 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:46.841077    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2203d97-4798-4cf0-84ce-94076c2568eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-llmhr\" (UID: \"b2203d97-4798-4cf0-84ce-94076c2568eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-llmhr"
Oct 30 17:53:46.841868 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:46.841847    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2203d97-4798-4cf0-84ce-94076c2568eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-llmhr\" (UID: \"b2203d97-4798-4cf0-84ce-94076c2568eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-llmhr"
Oct 30 17:53:46.843565 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:46.843536    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2203d97-4798-4cf0-84ce-94076c2568eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-llmhr\" (UID: \"b2203d97-4798-4cf0-84ce-94076c2568eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-llmhr"
Oct 30 17:53:46.864607 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:46.864582    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp9lg\" (UniqueName: \"kubernetes.io/projected/b2203d97-4798-4cf0-84ce-94076c2568eb-kube-api-access-rp9lg\") pod \"maas-api-revoke-sub-0165d7d2-llmhr\" (UID: \"b2203d97-4798-4cf0-84ce-94076c2568eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-llmhr"
Oct 30 17:53:46.886554 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:46.886510    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-llmhr"
Oct 30 17:53:47.190749 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:47.190715    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-llmhr"]
Oct 30 17:53:47.194607 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:53:47.194580    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2203d97_4798_4cf0_84ce_94076c2568eb.slice/crio-716d4cb0c4be8d15ed0689884e2465b25162dd8bb91235da41a914cac3f30f0a WatchSource:0}: Error finding container 716d4cb0c4be8d15ed0689884e2465b25162dd8bb91235da41a914cac3f30f0a: Status 404 returned error can't find the container with id 716d4cb0c4be8d15ed0689884e2465b25162dd8bb91235da41a914cac3f30f0a
Oct 30 17:53:47.606577 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:47.606532    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-llmhr" event={"ID":"b2203d97-4798-4cf0-84ce-94076c2568eb","Type":"ContainerStarted","Data":"98ee1ff5d303cd9d94847e41004606c5dc9e17a9854a50f721a757157d67ec16"}
Oct 30 17:53:47.606987 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:47.606605    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-llmhr" event={"ID":"b2203d97-4798-4cf0-84ce-94076c2568eb","Type":"ContainerStarted","Data":"716d4cb0c4be8d15ed0689884e2465b25162dd8bb91235da41a914cac3f30f0a"}
Oct 30 17:53:47.647711 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:47.647643    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-llmhr" podStartSLOduration=1.647625323 podStartE2EDuration="1.647625323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:53:47.646102735 +0000 UTC m=+4004.081004941" watchObservedRunningTime="2026-10-30 17:53:47.647625323 +0000 UTC m=+4004.082527530"
Oct 30 17:53:48.613142 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:48.613100    4433 generic.go:356] "Generic (PLEG): container finished" podID="b2203d97-4798-4cf0-84ce-94076c2568eb" containerID="98ee1ff5d303cd9d94847e41004606c5dc9e17a9854a50f721a757157d67ec16" exitCode=0
Oct 30 17:53:48.613569 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:48.613167    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-llmhr" event={"ID":"b2203d97-4798-4cf0-84ce-94076c2568eb","Type":"ContainerDied","Data":"98ee1ff5d303cd9d94847e41004606c5dc9e17a9854a50f721a757157d67ec16"}
Oct 30 17:53:49.753090 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:49.753058    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-llmhr"
Oct 30 17:53:49.871036 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:49.870935    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2203d97-4798-4cf0-84ce-94076c2568eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2203d97-4798-4cf0-84ce-94076c2568eb-maas-api-openshift-ca\") pod \"b2203d97-4798-4cf0-84ce-94076c2568eb\" (UID: \"b2203d97-4798-4cf0-84ce-94076c2568eb\") "
Oct 30 17:53:49.871227 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:49.871056    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2203d97-4798-4cf0-84ce-94076c2568eb-kube-api-access-rp9lg\" (UniqueName: \"kubernetes.io/projected/b2203d97-4798-4cf0-84ce-94076c2568eb-kube-api-access-rp9lg\") pod \"b2203d97-4798-4cf0-84ce-94076c2568eb\" (UID: \"b2203d97-4798-4cf0-84ce-94076c2568eb\") "
Oct 30 17:53:49.871227 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:49.871114    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2203d97-4798-4cf0-84ce-94076c2568eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2203d97-4798-4cf0-84ce-94076c2568eb-maas-api-xks-ca\") pod \"b2203d97-4798-4cf0-84ce-94076c2568eb\" (UID: \"b2203d97-4798-4cf0-84ce-94076c2568eb\") "
Oct 30 17:53:49.871416 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:49.871372    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2203d97-4798-4cf0-84ce-94076c2568eb-maas-api-openshift-ca" pod "b2203d97-4798-4cf0-84ce-94076c2568eb" (UID: "b2203d97-4798-4cf0-84ce-94076c2568eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:53:49.871585 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:49.871563    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2203d97-4798-4cf0-84ce-94076c2568eb-maas-api-openshift-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:53:49.873329 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:49.873299    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2203d97-4798-4cf0-84ce-94076c2568eb-kube-api-access-rp9lg" pod "b2203d97-4798-4cf0-84ce-94076c2568eb" (UID: "b2203d97-4798-4cf0-84ce-94076c2568eb"). InnerVolumeSpecName "kube-api-access-rp9lg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:53:49.873463 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:49.873305    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2203d97-4798-4cf0-84ce-94076c2568eb-maas-api-xks-ca" pod "b2203d97-4798-4cf0-84ce-94076c2568eb" (UID: "b2203d97-4798-4cf0-84ce-94076c2568eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:53:49.972762 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:49.972718    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp9lg\" (UniqueName: \"kubernetes.io/projected/b2203d97-4798-4cf0-84ce-94076c2568eb-kube-api-access-rp9lg\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:53:49.972762 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:49.972755    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2203d97-4798-4cf0-84ce-94076c2568eb-maas-api-xks-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:53:50.623334 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:50.623300    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-llmhr"
Oct 30 17:53:50.623334 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:50.623311    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-llmhr" event={"ID":"b2203d97-4798-4cf0-84ce-94076c2568eb","Type":"ContainerDied","Data":"716d4cb0c4be8d15ed0689884e2465b25162dd8bb91235da41a914cac3f30f0a"}
Oct 30 17:53:50.623589 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:50.623365    4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="716d4cb0c4be8d15ed0689884e2465b25162dd8bb91235da41a914cac3f30f0a"
Oct 30 17:53:56.976679 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:53:56.976637    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:53:57.552866 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:57.552783    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gp8mh"]
Oct 30 17:53:57.553211 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:57.553198    4433 cpu_manager.go:414] "containerMap: removing container" podUID="b2203d97-4798-4cf0-84ce-94076c2568eb" containerName="revoke-keys"
Oct 30 17:53:57.553259 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:57.553212    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2203d97-4798-4cf0-84ce-94076c2568eb" containerName="revoke-keys"
Oct 30 17:53:57.553296 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:57.553290    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2203d97-4798-4cf0-84ce-94076c2568eb" containerName="revoke-keys"
Oct 30 17:53:57.556354 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:57.556326    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gp8mh"
Oct 30 17:53:57.582096 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:57.582063    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 17:53:57.582279 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:57.582120    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:53:57.582279 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:57.582062    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 17:53:57.582279 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:57.582138    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\""
Oct 30 17:53:57.584267 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:57.584241    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gp8mh"]
Oct 30 17:53:57.640222 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:57.640189    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/642c65f2-8140-4877-a3ab-3561f491b245-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-gp8mh\" (UID: \"642c65f2-8140-4877-a3ab-3561f491b245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gp8mh"
Oct 30 17:53:57.640222 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:57.640223    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s528n\" (UniqueName: \"kubernetes.io/projected/642c65f2-8140-4877-a3ab-3561f491b245-kube-api-access-s528n\") pod \"maas-api-revoke-sub-6b19338f-gp8mh\" (UID: \"642c65f2-8140-4877-a3ab-3561f491b245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gp8mh"
Oct 30 17:53:57.640446 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:57.640249    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/642c65f2-8140-4877-a3ab-3561f491b245-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-gp8mh\" (UID: \"642c65f2-8140-4877-a3ab-3561f491b245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gp8mh"
Oct 30 17:53:57.741514 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:57.741479    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/642c65f2-8140-4877-a3ab-3561f491b245-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-gp8mh\" (UID: \"642c65f2-8140-4877-a3ab-3561f491b245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gp8mh"
Oct 30 17:53:57.741514 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:57.741516    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s528n\" (UniqueName: \"kubernetes.io/projected/642c65f2-8140-4877-a3ab-3561f491b245-kube-api-access-s528n\") pod \"maas-api-revoke-sub-6b19338f-gp8mh\" (UID: \"642c65f2-8140-4877-a3ab-3561f491b245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gp8mh"
Oct 30 17:53:57.741721 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:57.741543    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/642c65f2-8140-4877-a3ab-3561f491b245-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-gp8mh\" (UID: \"642c65f2-8140-4877-a3ab-3561f491b245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gp8mh"
Oct 30 17:53:57.742301 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:57.742278    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/642c65f2-8140-4877-a3ab-3561f491b245-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-gp8mh\" (UID: \"642c65f2-8140-4877-a3ab-3561f491b245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gp8mh"
Oct 30 17:53:57.743909 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:57.743877    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/642c65f2-8140-4877-a3ab-3561f491b245-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-gp8mh\" (UID: \"642c65f2-8140-4877-a3ab-3561f491b245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gp8mh"
Oct 30 17:53:57.764143 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:57.764118    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s528n\" (UniqueName: \"kubernetes.io/projected/642c65f2-8140-4877-a3ab-3561f491b245-kube-api-access-s528n\") pod \"maas-api-revoke-sub-6b19338f-gp8mh\" (UID: \"642c65f2-8140-4877-a3ab-3561f491b245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gp8mh"
Oct 30 17:53:57.869147 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:57.869068    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gp8mh"
Oct 30 17:53:58.168557 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:58.168504    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gp8mh"]
Oct 30 17:53:58.172549 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:53:58.172517    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod642c65f2_8140_4877_a3ab_3561f491b245.slice/crio-93bf4fd212364e0cec6d698166cca1e2a017287005fd8bc5ea32786cc803ba39 WatchSource:0}: Error finding container 93bf4fd212364e0cec6d698166cca1e2a017287005fd8bc5ea32786cc803ba39: Status 404 returned error can't find the container with id 93bf4fd212364e0cec6d698166cca1e2a017287005fd8bc5ea32786cc803ba39
Oct 30 17:53:58.660147 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:58.660112    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gp8mh" event={"ID":"642c65f2-8140-4877-a3ab-3561f491b245","Type":"ContainerStarted","Data":"6fa65ced09621b37d618db5f9397213c3b441f2421d9d72d81ab172f5ad10722"}
Oct 30 17:53:58.660323 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:58.660168    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gp8mh" event={"ID":"642c65f2-8140-4877-a3ab-3561f491b245","Type":"ContainerStarted","Data":"93bf4fd212364e0cec6d698166cca1e2a017287005fd8bc5ea32786cc803ba39"}
Oct 30 17:53:58.700866 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:58.700807    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gp8mh" podStartSLOduration=1.7007933739999999 podStartE2EDuration="1.700793374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:53:58.699314829 +0000 UTC m=+4015.134217052" watchObservedRunningTime="2026-10-30 17:53:58.700793374 +0000 UTC m=+4015.135695585"
Oct 30 17:53:59.666083 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:59.666044    4433 generic.go:356] "Generic (PLEG): container finished" podID="642c65f2-8140-4877-a3ab-3561f491b245" containerID="6fa65ced09621b37d618db5f9397213c3b441f2421d9d72d81ab172f5ad10722" exitCode=0
Oct 30 17:53:59.666580 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:53:59.666132    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gp8mh" event={"ID":"642c65f2-8140-4877-a3ab-3561f491b245","Type":"ContainerDied","Data":"6fa65ced09621b37d618db5f9397213c3b441f2421d9d72d81ab172f5ad10722"}
Oct 30 17:54:00.813792 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:00.813763    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gp8mh"
Oct 30 17:54:00.870146 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:00.870114    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/642c65f2-8140-4877-a3ab-3561f491b245-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/642c65f2-8140-4877-a3ab-3561f491b245-maas-api-openshift-ca\") pod \"642c65f2-8140-4877-a3ab-3561f491b245\" (UID: \"642c65f2-8140-4877-a3ab-3561f491b245\") "
Oct 30 17:54:00.870333 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:00.870227    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/642c65f2-8140-4877-a3ab-3561f491b245-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/642c65f2-8140-4877-a3ab-3561f491b245-maas-api-xks-ca\") pod \"642c65f2-8140-4877-a3ab-3561f491b245\" (UID: \"642c65f2-8140-4877-a3ab-3561f491b245\") "
Oct 30 17:54:00.870333 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:00.870250    4433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/642c65f2-8140-4877-a3ab-3561f491b245-kube-api-access-s528n\" (UniqueName: \"kubernetes.io/projected/642c65f2-8140-4877-a3ab-3561f491b245-kube-api-access-s528n\") pod \"642c65f2-8140-4877-a3ab-3561f491b245\" (UID: \"642c65f2-8140-4877-a3ab-3561f491b245\") "
Oct 30 17:54:00.870570 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:00.870546    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/642c65f2-8140-4877-a3ab-3561f491b245-maas-api-openshift-ca" pod "642c65f2-8140-4877-a3ab-3561f491b245" (UID: "642c65f2-8140-4877-a3ab-3561f491b245"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:54:00.872312 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:00.872285    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/642c65f2-8140-4877-a3ab-3561f491b245-maas-api-xks-ca" pod "642c65f2-8140-4877-a3ab-3561f491b245" (UID: "642c65f2-8140-4877-a3ab-3561f491b245"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:54:00.872444 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:00.872363    4433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/642c65f2-8140-4877-a3ab-3561f491b245-kube-api-access-s528n" pod "642c65f2-8140-4877-a3ab-3561f491b245" (UID: "642c65f2-8140-4877-a3ab-3561f491b245"). InnerVolumeSpecName "kube-api-access-s528n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:54:00.971542 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:00.971508    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/642c65f2-8140-4877-a3ab-3561f491b245-maas-api-openshift-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:54:00.971542 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:00.971538    4433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/642c65f2-8140-4877-a3ab-3561f491b245-maas-api-xks-ca\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:54:00.971542 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:00.971547    4433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s528n\" (UniqueName: \"kubernetes.io/projected/642c65f2-8140-4877-a3ab-3561f491b245-kube-api-access-s528n\") on node \"ip-10-0-131-13.ec2.internal\" DevicePath \"\""
Oct 30 17:54:01.676313 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:01.676275    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gp8mh" event={"ID":"642c65f2-8140-4877-a3ab-3561f491b245","Type":"ContainerDied","Data":"93bf4fd212364e0cec6d698166cca1e2a017287005fd8bc5ea32786cc803ba39"}
Oct 30 17:54:01.676562 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:01.676333    4433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93bf4fd212364e0cec6d698166cca1e2a017287005fd8bc5ea32786cc803ba39"
Oct 30 17:54:01.676562 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:01.676288    4433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gp8mh"
Oct 30 17:54:08.743796 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:08.743759    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mbzbg"]
Oct 30 17:54:20.484828 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:20.484794    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-6strf_24a380dc-cf10-49d0-a874-db00438648dc/manager/0.log"
Oct 30 17:54:20.921943 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:20.921844    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-54btj_93b3e2e5-f146-47a4-92b3-189bfdb8b080/manager/0.log"
Oct 30 17:54:21.517713 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:21.517684    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-nt54h_515a8205-3b3b-4e80-9b96-79c33acfdc06/server/0.log"
Oct 30 17:54:21.715886 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:21.715844    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-zhx25_4452411d-5b03-40ae-a3d3-e96adf966c46/manager/0.log"
Oct 30 17:54:21.942219 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:21.942171    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-j2dld_8ebb4433-b279-485f-b5c8-7b72cefaec9e/manager/0.log"
Oct 30 17:54:22.724814 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:22.724783    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68c46f97f5-wpvf2_f08eb4e1-c182-499c-8fa3-d8e57cdebaeb/copy-manifests/0.log"
Oct 30 17:54:22.757340 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:22.757312    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68c46f97f5-wpvf2_f08eb4e1-c182-499c-8fa3-d8e57cdebaeb/manager/0.log"
Oct 30 17:54:24.258214 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:24.258182    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-llmhr_b2203d97-4798-4cf0-84ce-94076c2568eb/revoke-keys/0.log"
Oct 30 17:54:25.003977 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:25.003952    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-gp8mh_642c65f2-8140-4877-a3ab-3561f491b245/revoke-keys/0.log"
Oct 30 17:54:25.772778 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:25.772746    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-njf4t_80afb90a-700b-4b28-ab7b-e649e5c9bd01/revoke-keys/0.log"
Oct 30 17:54:26.977685 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:54:26.977643    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:54:27.779940 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:27.779901    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-428zq_7c756c06-62e5-449c-8ed7-d17d45c33839/manager/0.log"
Oct 30 17:54:28.710484 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:28.710450    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-mbzbg_f6e2dab6-5a9d-49a1-ad32-1680c0d2af56/limitador/0.log"
Oct 30 17:54:29.723097 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:29.723057    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c85d847d5njlxb_edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a/istio-proxy/0.log"
Oct 30 17:54:29.928327 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:29.928292    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-7pp9d_bf2a3652-49a3-4e6e-b319-562eea16f39e/discovery/0.log"
Oct 30 17:54:30.939327 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:30.939297    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-dfcvz_bef99f8d-c7bf-4206-960a-50c84ec6d79d/istio-proxy/0.log"
Oct 30 17:54:31.331106 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:31.331021    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-jpwg6_30c47f8e-91a0-4b8c-8689-8eb97f69cc7a/istio-proxy/0.log"
Oct 30 17:54:31.723566 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:31.723536    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-n9gd9_c986e3ae-ffab-420f-92ca-8234d2d918f6/istio-proxy/0.log"
Oct 30 17:54:45.827280 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:45.827249    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-6strf_24a380dc-cf10-49d0-a874-db00438648dc/manager/0.log"
Oct 30 17:54:46.265658 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:46.265622    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-54btj_93b3e2e5-f146-47a4-92b3-189bfdb8b080/manager/0.log"
Oct 30 17:54:46.755533 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:46.755502    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-nt54h_515a8205-3b3b-4e80-9b96-79c33acfdc06/server/0.log"
Oct 30 17:54:46.940304 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:46.940268    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-zhx25_4452411d-5b03-40ae-a3d3-e96adf966c46/manager/0.log"
Oct 30 17:54:47.173578 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:47.173498    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-j2dld_8ebb4433-b279-485f-b5c8-7b72cefaec9e/manager/0.log"
Oct 30 17:54:47.958229 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:47.958199    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68c46f97f5-wpvf2_f08eb4e1-c182-499c-8fa3-d8e57cdebaeb/manager/0.log"
Oct 30 17:54:47.988529 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:47.988502    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68c46f97f5-wpvf2_f08eb4e1-c182-499c-8fa3-d8e57cdebaeb/copy-manifests/0.log"
Oct 30 17:54:49.142716 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:49.142683    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-llmhr_b2203d97-4798-4cf0-84ce-94076c2568eb/revoke-keys/0.log"
Oct 30 17:54:49.889861 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:49.889824    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-gp8mh_642c65f2-8140-4877-a3ab-3561f491b245/revoke-keys/0.log"
Oct 30 17:54:50.629483 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:50.629452    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-njf4t_80afb90a-700b-4b28-ab7b-e649e5c9bd01/revoke-keys/0.log"
Oct 30 17:54:52.287125 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:52.287089    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-428zq_7c756c06-62e5-449c-8ed7-d17d45c33839/manager/0.log"
Oct 30 17:54:53.186929 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:53.186899    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-mbzbg_f6e2dab6-5a9d-49a1-ad32-1680c0d2af56/limitador/0.log"
Oct 30 17:54:53.968787 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:53.968723    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c85d847d5njlxb_edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a/istio-proxy/0.log"
Oct 30 17:54:54.183572 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:54.183538    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-7pp9d_bf2a3652-49a3-4e6e-b319-562eea16f39e/discovery/0.log"
Oct 30 17:54:55.195724 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:55.195686    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-dfcvz_bef99f8d-c7bf-4206-960a-50c84ec6d79d/istio-proxy/0.log"
Oct 30 17:54:55.580159 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:55.580058    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-jpwg6_30c47f8e-91a0-4b8c-8689-8eb97f69cc7a/istio-proxy/0.log"
Oct 30 17:54:55.972362 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:54:55.972327    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-n9gd9_c986e3ae-ffab-420f-92ca-8234d2d918f6/istio-proxy/0.log"
Oct 30 17:54:56.977306 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:54:56.977271    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:55:26.976362 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:55:26.976320    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:55:56.976682 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:55:56.976641    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:56:17.452783 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:17.452751    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5cgn8_498ef897-364c-4e19-9fbf-cdc46aac623f/global-pull-secret-syncer/0.log"
Oct 30 17:56:18.616510 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:18.616464    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vw5v7_6c03c304-7756-45bb-9597-350026ed361e/konnectivity-agent/0.log"
Oct 30 17:56:18.851270 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:18.851236    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-13.ec2.internal_777e8481b0741b0fafcaf01b060f8274/haproxy/0.log"
Oct 30 17:56:25.207607 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:25.207579    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-428zq_7c756c06-62e5-449c-8ed7-d17d45c33839/manager/0.log"
Oct 30 17:56:25.766544 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:25.766513    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-mbzbg_f6e2dab6-5a9d-49a1-ad32-1680c0d2af56/limitador/0.log"
Oct 30 17:56:26.977835 ip-10-0-131-13 kubenswrapper[4433]: E1030 17:56:26.977797    4433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mrllk" containerName="extractor"
Oct 30 17:56:27.269005 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:27.268919    4433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2tldp/perf-node-gather-daemonset-mlpz7"]
Oct 30 17:56:27.269539 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:27.269518    4433 cpu_manager.go:414] "containerMap: removing container" podUID="642c65f2-8140-4877-a3ab-3561f491b245" containerName="revoke-keys"
Oct 30 17:56:27.269633 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:27.269540    4433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="642c65f2-8140-4877-a3ab-3561f491b245" containerName="revoke-keys"
Oct 30 17:56:27.269633 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:27.269624    4433 memory_manager.go:365] "RemoveStaleState removing state" podUID="642c65f2-8140-4877-a3ab-3561f491b245" containerName="revoke-keys"
Oct 30 17:56:27.272711 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:27.272693    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-mlpz7"
Oct 30 17:56:27.291555 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:27.291520    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2tldp/perf-node-gather-daemonset-mlpz7"]
Oct 30 17:56:27.315387 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:27.315357    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2tldp\"/\"openshift-service-ca.crt\""
Oct 30 17:56:27.315387 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:27.315366    4433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2tldp\"/\"kube-root-ca.crt\""
Oct 30 17:56:27.380227 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:27.380197    4433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2tldp\"/\"default-dockercfg-t845d\""
Oct 30 17:56:27.399779 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:27.399749    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/84024064-03cb-4306-b2f9-0f13e017fa16-podres\") pod \"perf-node-gather-daemonset-mlpz7\" (UID: \"84024064-03cb-4306-b2f9-0f13e017fa16\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-mlpz7"
Oct 30 17:56:27.399779 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:27.399782    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/84024064-03cb-4306-b2f9-0f13e017fa16-proc\") pod \"perf-node-gather-daemonset-mlpz7\" (UID: \"84024064-03cb-4306-b2f9-0f13e017fa16\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-mlpz7"
Oct 30 17:56:27.399950 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:27.399814    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84024064-03cb-4306-b2f9-0f13e017fa16-sys\") pod \"perf-node-gather-daemonset-mlpz7\" (UID: \"84024064-03cb-4306-b2f9-0f13e017fa16\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-mlpz7"
Oct 30 17:56:27.399950 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:27.399888    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvbnf\" (UniqueName: \"kubernetes.io/projected/84024064-03cb-4306-b2f9-0f13e017fa16-kube-api-access-cvbnf\") pod \"perf-node-gather-daemonset-mlpz7\" (UID: \"84024064-03cb-4306-b2f9-0f13e017fa16\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-mlpz7"
Oct 30 17:56:27.400026 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:27.399961    4433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/84024064-03cb-4306-b2f9-0f13e017fa16-lib-modules\") pod \"perf-node-gather-daemonset-mlpz7\" (UID: \"84024064-03cb-4306-b2f9-0f13e017fa16\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-mlpz7"
Oct 30 17:56:27.500951 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:27.500904    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/84024064-03cb-4306-b2f9-0f13e017fa16-podres\") pod \"perf-node-gather-daemonset-mlpz7\" (UID: \"84024064-03cb-4306-b2f9-0f13e017fa16\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-mlpz7"
Oct 30 17:56:27.500951 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:27.500952    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/84024064-03cb-4306-b2f9-0f13e017fa16-proc\") pod \"perf-node-gather-daemonset-mlpz7\" (UID: \"84024064-03cb-4306-b2f9-0f13e017fa16\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-mlpz7"
Oct 30 17:56:27.501220 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:27.500995    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84024064-03cb-4306-b2f9-0f13e017fa16-sys\") pod \"perf-node-gather-daemonset-mlpz7\" (UID: \"84024064-03cb-4306-b2f9-0f13e017fa16\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-mlpz7"
Oct 30 17:56:27.501220 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:27.501021    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvbnf\" (UniqueName: \"kubernetes.io/projected/84024064-03cb-4306-b2f9-0f13e017fa16-kube-api-access-cvbnf\") pod \"perf-node-gather-daemonset-mlpz7\" (UID: \"84024064-03cb-4306-b2f9-0f13e017fa16\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-mlpz7"
Oct 30 17:56:27.501220 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:27.501050    4433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/84024064-03cb-4306-b2f9-0f13e017fa16-lib-modules\") pod \"perf-node-gather-daemonset-mlpz7\" (UID: \"84024064-03cb-4306-b2f9-0f13e017fa16\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-mlpz7"
Oct 30 17:56:27.501220 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:27.501049    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/84024064-03cb-4306-b2f9-0f13e017fa16-podres\") pod \"perf-node-gather-daemonset-mlpz7\" (UID: \"84024064-03cb-4306-b2f9-0f13e017fa16\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-mlpz7"
Oct 30 17:56:27.501220 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:27.501104    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/84024064-03cb-4306-b2f9-0f13e017fa16-proc\") pod \"perf-node-gather-daemonset-mlpz7\" (UID: \"84024064-03cb-4306-b2f9-0f13e017fa16\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-mlpz7"
Oct 30 17:56:27.501220 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:27.501114    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84024064-03cb-4306-b2f9-0f13e017fa16-sys\") pod \"perf-node-gather-daemonset-mlpz7\" (UID: \"84024064-03cb-4306-b2f9-0f13e017fa16\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-mlpz7"
Oct 30 17:56:27.501220 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:27.501193    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/84024064-03cb-4306-b2f9-0f13e017fa16-lib-modules\") pod \"perf-node-gather-daemonset-mlpz7\" (UID: \"84024064-03cb-4306-b2f9-0f13e017fa16\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-mlpz7"
Oct 30 17:56:27.523749 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:27.523681    4433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvbnf\" (UniqueName: \"kubernetes.io/projected/84024064-03cb-4306-b2f9-0f13e017fa16-kube-api-access-cvbnf\") pod \"perf-node-gather-daemonset-mlpz7\" (UID: \"84024064-03cb-4306-b2f9-0f13e017fa16\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-mlpz7"
Oct 30 17:56:27.584809 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:27.584771    4433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-mlpz7"
Oct 30 17:56:27.880304 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:27.880220    4433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2tldp/perf-node-gather-daemonset-mlpz7"]
Oct 30 17:56:27.883318 ip-10-0-131-13 kubenswrapper[4433]: W1030 17:56:27.883290    4433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod84024064_03cb_4306_b2f9_0f13e017fa16.slice/crio-71005bbfccb44b72c1fd8ad8b840cdd63efbc31833e52dd47387a7b8c968d68b WatchSource:0}: Error finding container 71005bbfccb44b72c1fd8ad8b840cdd63efbc31833e52dd47387a7b8c968d68b: Status 404 returned error can't find the container with id 71005bbfccb44b72c1fd8ad8b840cdd63efbc31833e52dd47387a7b8c968d68b
Oct 30 17:56:28.376511 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:28.376475    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-mlpz7" event={"ID":"84024064-03cb-4306-b2f9-0f13e017fa16","Type":"ContainerStarted","Data":"17948e7356bced4b9189f91c8ee68d0ee8288af39cba275b1a372fa3ac14ed26"}
Oct 30 17:56:28.376981 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:28.376525    4433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-mlpz7" event={"ID":"84024064-03cb-4306-b2f9-0f13e017fa16","Type":"ContainerStarted","Data":"71005bbfccb44b72c1fd8ad8b840cdd63efbc31833e52dd47387a7b8c968d68b"}
Oct 30 17:56:28.376981 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:28.376624    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-mlpz7"
Oct 30 17:56:28.417054 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:28.416994    4433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-mlpz7" podStartSLOduration=1.4169793529999999 podStartE2EDuration="1.416979353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:28.416793532 +0000 UTC m=+4164.851695740" watchObservedRunningTime="2026-10-30 17:56:28.416979353 +0000 UTC m=+4164.851881577"
Oct 30 17:56:28.589366 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:28.589336    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-854dcbf887-h7wp4_44f1b4ab-6c59-47e3-9c20-d3c6cda4f21c/metrics-server/0.log"
Oct 30 17:56:28.709548 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:28.709518    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-bbb4g_215e894c-1ace-4564-a694-d535a1cffeba/monitoring-plugin/0.log"
Oct 30 17:56:28.831490 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:28.831458    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cv7gw_7e04ca71-89c2-4232-9da2-7034affb6a99/node-exporter/0.log"
Oct 30 17:56:28.945549 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:28.945519    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cv7gw_7e04ca71-89c2-4232-9da2-7034affb6a99/kube-rbac-proxy/0.log"
Oct 30 17:56:29.060016 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:29.059940    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cv7gw_7e04ca71-89c2-4232-9da2-7034affb6a99/init-textfile/0.log"
Oct 30 17:56:30.281382 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:30.281355    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e10cae9c-9374-4615-8eda-d5d8d62173a3/prometheus/0.log"
Oct 30 17:56:30.395561 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:30.395533    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e10cae9c-9374-4615-8eda-d5d8d62173a3/config-reloader/0.log"
Oct 30 17:56:30.512088 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:30.512059    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e10cae9c-9374-4615-8eda-d5d8d62173a3/thanos-sidecar/0.log"
Oct 30 17:56:30.625713 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:30.625638    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e10cae9c-9374-4615-8eda-d5d8d62173a3/kube-rbac-proxy-web/0.log"
Oct 30 17:56:30.739780 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:30.739754    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e10cae9c-9374-4615-8eda-d5d8d62173a3/kube-rbac-proxy/0.log"
Oct 30 17:56:30.859431 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:30.859379    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e10cae9c-9374-4615-8eda-d5d8d62173a3/kube-rbac-proxy-thanos/0.log"
Oct 30 17:56:30.973736 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:30.973711    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e10cae9c-9374-4615-8eda-d5d8d62173a3/init-config-reloader/0.log"
Oct 30 17:56:31.099859 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:31.099826    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-xd2jw_5a07cafb-21a7-4ad5-ab8d-34ee4d6d1394/prometheus-operator/0.log"
Oct 30 17:56:31.208645 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:31.208606    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-xd2jw_5a07cafb-21a7-4ad5-ab8d-34ee4d6d1394/kube-rbac-proxy/0.log"
Oct 30 17:56:31.324185 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:31.324110    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-cxb62_ddfadf21-5be2-439c-af86-4c7ccf9545e7/prometheus-operator-admission-webhook/0.log"
Oct 30 17:56:31.798213 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:31.798181    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c8f575fb-jsx25_52a9edcf-c077-4650-844b-24a3c6a0f87e/thanos-query/0.log"
Oct 30 17:56:31.912015 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:31.911988    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c8f575fb-jsx25_52a9edcf-c077-4650-844b-24a3c6a0f87e/kube-rbac-proxy-web/0.log"
Oct 30 17:56:32.028935 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:32.028909    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c8f575fb-jsx25_52a9edcf-c077-4650-844b-24a3c6a0f87e/kube-rbac-proxy/0.log"
Oct 30 17:56:32.143434 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:32.143336    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c8f575fb-jsx25_52a9edcf-c077-4650-844b-24a3c6a0f87e/prom-label-proxy/0.log"
Oct 30 17:56:32.258636 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:32.258605    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c8f575fb-jsx25_52a9edcf-c077-4650-844b-24a3c6a0f87e/kube-rbac-proxy-rules/0.log"
Oct 30 17:56:32.373507 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:32.373479    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c8f575fb-jsx25_52a9edcf-c077-4650-844b-24a3c6a0f87e/kube-rbac-proxy-metrics/0.log"
Oct 30 17:56:34.392529 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:34.392503    4433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-mlpz7"
Oct 30 17:56:36.053093 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:36.053057    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-rtst9_dd9cd8fd-063f-4f87-a80c-57f65f0d220b/download-server/0.log"
Oct 30 17:56:39.513189 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:39.513157    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5ztj2_03a9bc00-f973-4d50-ac32-a4ca843b977b/dns/0.log"
Oct 30 17:56:39.624578 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:39.624542    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5ztj2_03a9bc00-f973-4d50-ac32-a4ca843b977b/kube-rbac-proxy/0.log"
Oct 30 17:56:40.502668 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:40.502634    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nlr68_92b17f34-17e7-4d4f-b43a-4e828ede0187/dns-node-resolver/0.log"
Oct 30 17:56:40.639483 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:40.639446    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c85d847d5njlxb_edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a/istio-proxy/0.log"
Oct 30 17:56:41.907496 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:41.907461    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jhqzq_db037093-50f6-49b9-93c2-f37a7f1290d6/node-ca/0.log"
Oct 30 17:56:42.529613 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:42.529571    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-7pp9d_bf2a3652-49a3-4e6e-b319-562eea16f39e/discovery/0.log"
Oct 30 17:56:42.843137 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:42.843058    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-dfcvz_bef99f8d-c7bf-4206-960a-50c84ec6d79d/istio-proxy/0.log"
Oct 30 17:56:42.975608 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:42.975577    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-jpwg6_30c47f8e-91a0-4b8c-8689-8eb97f69cc7a/istio-proxy/0.log"
Oct 30 17:56:43.110100 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:43.110019    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-n9gd9_c986e3ae-ffab-420f-92ca-8234d2d918f6/istio-proxy/0.log"
Oct 30 17:56:43.666539 ip-10-0-131-13 kubenswrapper[4433]: I1030 17:56:43.666492    4433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c85d847d5njlxb_edfc7ebb-fa0e-44fe-b442-f50ce4e9aa2a/istio-proxy/0.log"